Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 1485428 Details for
Bug 1631667
[free-stg] write error "file already closed" for prometheus container
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
prometheus container logs
prometheus.txt (text/plain), 7.25 MB, created by
Junqi Zhao
on 2018-09-21 08:51:28 UTC
(
hide
)
Description:
prometheus container logs
Filename:
MIME Type:
Creator:
Junqi Zhao
Created:
2018-09-21 08:51:28 UTC
Size:
7.25 MB
patch
obsolete
>level=warn ts=2018-09-21T04:31:32.90552943Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:33.226148324Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:33.226484141Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:33.795312271Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:33.796053293Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:33.818427252Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:33.819722801Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:33.822057277Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:34.133789893Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:34.134034359Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:35.308533816Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:35.57669384Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:35.577115539Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:36.656677594Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:36.657098492Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:36.926500885Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:37.169754891Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:37.170201936Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:39.488222462Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:39.489180162Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:40.687537927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:40.687901021Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:31:40.859459438Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:41.250567561Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:41.250858787Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:41.487494792Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:41.518759208Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:41.519116672Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:41.559450101Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:41.559926373Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:41.709056671Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:42.211758118Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:42.212190255Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:43.865940833Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:43.998153826Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:44.339168671Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:45.220070591Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:45.22038307Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:45.358097875Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:45.358859443Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:45.764482993Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:45.765204283Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:45.922290709Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:45.92290399Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:46.375381106Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:46.598749799Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:46.599094761Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:46.983080606Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:47.22429141Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:47.224602326Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:48.259065363Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:48.554637755Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:48.651210807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:48.65161743Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:49.203740735Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:49.505269395Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:49.505568344Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:49.692604451Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:31:50.859456697Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:51.709183973Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:51.710074526Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:51.899195551Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:51.899868897Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:51.985525962Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:52.084339418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:52.084785533Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:52.577886744Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:52.691180458Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:52.999512169Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:53.000305023Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:53.346905209Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:53.896829465Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:54.973206431Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:54.973387995Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:55.35052416Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:55.350985774Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:55.429335968Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:55.42947173Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:55.750331394Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:55.750804534Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:56.403069993Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:58.263729949Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:58.855814237Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:59.284991876Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:59.285378019Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:59.712827579Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:59.713078407Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:59.888322545Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:31:59.888484145Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:32:00.859423057Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:00.935338698Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:01.005777728Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:01.006310416Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:01.683708335Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:02.357327396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:02.357479823Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:02.675374448Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:02.676134047Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:03.699276065Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:04.678302526Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:04.724168527Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:07.171059092Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:07.200021109Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:07.218574649Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:07.218787635Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:07.256049453Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:07.256824566Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:08.025373882Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:08.450242621Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:09.520174598Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:09.522038782Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:09.691145733Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:09.692276334Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:10.037135951Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:10.463277967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:32:10.86280707Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:12.441781517Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:12.442325316Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:12.520562222Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:14.841154522Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:16.840336413Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:16.840815147Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:17.073840198Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:17.426022167Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:17.426417727Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:18.177901019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:19.409260886Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:20.009817593Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:20.01053633Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:32:20.859421829Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:21.20891294Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:21.209254676Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:21.903976268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:21.904105936Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:22.772525679Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:23.116397995Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:23.116620489Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:25.697069709Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:25.726149073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:25.726398424Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:26.271832299Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:26.272102874Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:27.378260293Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:27.378537601Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:28.160099455Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:28.16072859Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:28.594845675Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:28.595366759Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:28.940083311Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:32:30.85943032Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:31.769438851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:31.769712004Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:32.282671468Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:32.283094647Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:32.35000268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:32.350356876Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:32.435261078Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:32.435802393Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:32.564480271Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:32.564887526Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:32.905699549Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:33.214016752Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:33.214337031Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:33.795766117Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:33.796712464Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:33.82401437Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:33.825581838Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:33.828620669Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:34.133370023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:34.133611942Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:35.310381352Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:35.577226286Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:35.57764023Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:36.604394063Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:36.604780134Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:36.92513644Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:37.1609878Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:37.161447588Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:39.268259897Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:39.269312816Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:40.701046294Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:40.70128441Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:32:40.859447492Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:41.242885125Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:41.243202533Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:41.486839596Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:41.509308575Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:41.509684151Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:41.556342207Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:41.5566432Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:41.712279603Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:42.544895822Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:42.545320114Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:43.952446395Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:43.997834195Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:44.338909088Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:45.125023992Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:45.125326916Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:45.360361679Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:45.361781206Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:45.738944922Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:45.739602668Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:45.923344196Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:45.923993293Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:46.376114432Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:46.653316487Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:46.653694709Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:46.985443863Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:47.059938813Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:47.060245567Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:48.267104018Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:48.510717532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:48.511190009Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:48.553410349Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:49.208172923Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:49.492837993Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:49.493087444Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:49.731018428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:32:50.859432276Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:51.70711492Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:51.707848857Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:51.905461989Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:51.905947571Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:52.017316142Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:52.069588613Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:52.070011948Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:52.579059411Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:52.693556061Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:53.000249828Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:53.000937227Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:53.333184489Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:53.897879624Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:54.975052097Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:54.975255074Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:55.332199111Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:55.332710664Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:55.429972862Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:55.430117207Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:55.791885354Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:55.792331138Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:56.410307377Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:58.261611505Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:58.858375473Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:59.291285515Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:59.291574926Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:59.71417915Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:59.714468842Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:59.887248419Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:32:59.887401374Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:33:00.859436798Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:00.930515618Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:01.046377356Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:01.046859613Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:01.649453374Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:02.345553052Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:02.345729879Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:02.678886376Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:02.679735039Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:03.701555458Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:04.67901303Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:04.721651887Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:07.167935224Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:07.197073614Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:07.215875928Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:07.216131026Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:07.28448453Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:07.285307797Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:08.013875915Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:08.163178353Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:08.850753284Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:08.851285131Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:09.929439572Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:09.930384699Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:10.018731409Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:10.451167881Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:33:10.859438401Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:12.432356982Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:12.433431501Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:12.595285861Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:14.841085648Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:16.686302205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:16.686602098Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:16.841759644Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:16.842207628Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:17.075109927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:18.184375241Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:19.407759878Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:19.688038193Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:19.688535291Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:33:20.859436835Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:21.206947968Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:21.20727288Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:21.933774832Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:21.933965367Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:22.761565743Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:23.115903186Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:23.116183534Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:25.699123081Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:25.727621756Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:25.72798968Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:26.272008773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:26.272293808Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:28.159983709Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:28.160530046Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:28.7924584Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:28.793245253Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:28.933100064Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:33:30.859421485Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:31.652755816Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:31.653250734Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:31.769519315Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:31.769776961Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:32.279780324Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:32.280187926Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:32.360326352Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:32.360697258Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:32.592544741Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:32.592980325Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:32.90551959Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:33.214904161Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:33.215247801Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:33.795338172Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:33.796087047Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:33.819647022Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:33.821222921Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:33.823943858Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:34.132811467Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:34.133050216Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:34.66019774Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:34.660498966Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:35.311723314Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:35.577384361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:35.577976631Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:36.604562412Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:36.604981357Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:36.921185435Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:37.160805031Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:37.161221469Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:39.208424582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:39.209435997Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:40.687843843Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:40.688161576Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:33:40.859437208Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:41.048565145Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:41.048907954Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:41.488576715Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:41.508433388Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:41.509006359Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:41.556252583Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:41.55658484Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:41.721886049Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:42.551965404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:42.552385427Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:43.855346171Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:44.006399661Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:44.337704908Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:44.556802683Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:44.557125041Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:45.358113535Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:45.359403457Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:45.718790032Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:45.719442496Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:45.914152528Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:45.914808891Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:46.374985324Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:46.678189548Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:46.678638473Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:46.984995524Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:47.060645747Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:47.060984117Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:48.253606847Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:48.520559193Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:48.521068145Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:48.586901943Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:49.202988549Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:49.492968887Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:49.493212174Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:49.640654122Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:33:50.859424174Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:51.701884333Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:51.702557637Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:51.906241899Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:51.906781868Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:51.979703819Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:52.074163627Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:52.074577401Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:52.577717351Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:52.691570729Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:52.998918526Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:52.999544663Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:53.328763789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:53.897512843Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:54.980688512Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:54.9808986Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:55.304871743Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:55.305305989Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:55.412901536Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:55.413061705Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:55.739143122Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:55.73957152Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:56.402013964Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:58.271897063Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:58.85339388Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:59.294099659Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:59.294385091Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:59.713016286Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:59.713300583Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:59.870377429Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:33:59.870502772Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:34:00.859429582Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:00.93019222Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:01.018026933Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:01.018592123Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:01.817628425Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:02.383720807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:02.383879604Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:02.728763708Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:02.72944263Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:03.70253467Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:04.678980817Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:04.720693996Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:07.175813572Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:07.205881903Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:07.226091962Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:07.226346999Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:07.279167562Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:07.279990084Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:08.012473665Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:08.469227472Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:08.803962172Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:08.804346779Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:09.747566391Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:09.748284808Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:10.033749439Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:10.434917862Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:34:10.859454969Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:12.493768328Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:12.495089911Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:12.596313617Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:14.843146704Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:16.823741633Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:16.824184629Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:17.074332588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:17.532342513Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:17.532730753Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:18.232565191Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:19.40847561Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:20.005009Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:20.005537105Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:34:20.859428936Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:21.216039803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:21.216387212Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:21.870445956Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:21.870621876Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:22.774243407Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:23.11623884Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:23.116498004Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:25.702955716Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:25.727096621Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:25.727341418Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:26.343736428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:26.344012845Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:27.378173343Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:27.378442751Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:28.165813276Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:28.166409237Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:28.716265939Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:28.717772707Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:28.935509835Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:34:30.859455456Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:31.650615434Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:31.65172877Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:31.769874525Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:31.770205941Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:31.957011166Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:31.957435121Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:32.464963309Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:32.465440705Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:32.5996009Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:32.600201108Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:32.905835912Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:33.217254514Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:33.217760566Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:33.795550872Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:33.796284586Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:33.815771756Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:33.816857693Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:33.81900481Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:34.132487284Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:34.132762124Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:35.309576818Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:35.577016181Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:35.577478579Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:36.64350842Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:36.643924197Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:36.925391984Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:37.175747402Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:37.176196311Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:39.303954271Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:39.304427336Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:40.685815268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:40.686061834Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:34:40.859426885Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:41.257427316Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:41.257733913Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:41.424847165Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:41.425248139Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:41.518115442Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:41.568692506Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:41.569025905Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:41.715452789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:42.540811538Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:42.541226731Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:43.861729139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:43.998990353Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:44.338153126Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:45.117199563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:45.11749143Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:45.358067444Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:45.358795926Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:45.72939845Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:45.730103129Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:45.926454685Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:45.927090647Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:46.375907117Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:46.776504997Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:46.776939086Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:46.980586217Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:47.061429531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:47.061728047Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:48.256960396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:48.511979921Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:48.512560725Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:48.55530183Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:49.200006448Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:49.576904744Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:49.577218708Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:49.651474038Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:34:50.859450177Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:51.714259942Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:51.715227498Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:51.917033777Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:51.917725004Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:51.994049145Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:52.077415322Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:52.077856549Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:52.578121081Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:52.690959996Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:52.999779189Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:53.000303747Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:53.326447018Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:53.898350924Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:54.97282548Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:54.973036945Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:55.29948606Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:55.299965357Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:55.412539567Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:55.412718195Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:55.734399783Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:55.734846272Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:56.4028382Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:58.260822533Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:58.854289869Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:59.305063486Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:59.305357653Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:59.712252012Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:59.712523638Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:59.870761002Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:34:59.870924239Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:35:00.85941829Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:00.934982408Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:00.936359959Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:00.936867141Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:02.574023507Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:02.576311093Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:02.738877079Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:02.846863482Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:02.847573354Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:03.698343489Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:04.682405826Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:04.721063845Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:07.17474143Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:07.206618606Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:07.225601197Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:07.225852159Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:07.252220153Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:07.252969913Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:08.017578169Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:08.152386015Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:09.439957475Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:09.440263019Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:09.691421437Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:09.691904935Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:10.028567889Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:10.492535527Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:35:10.864522192Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:12.434895145Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:12.435477501Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:12.516481778Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:14.840876315Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:16.719630479Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:16.720116087Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:17.077373313Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:17.482966148Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:17.483282236Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:18.171241783Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:19.409265659Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:19.707432342Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:19.707947164Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:35:20.859509637Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:21.207112316Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:21.207469179Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:21.916337542Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:21.916490569Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:22.77745743Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:23.115587745Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:23.115852642Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:25.697410403Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:25.726995214Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:25.727295085Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:26.274203609Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:26.274463568Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:27.384622582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:27.384926164Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:28.179784692Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:28.180386277Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:28.692626033Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:28.693190546Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:28.944607114Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:35:30.859419175Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:31.61737333Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:31.6179648Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:31.769080882Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:31.769309909Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:31.970402136Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:31.97082939Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:32.444407725Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:32.444774907Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:32.599500004Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:32.600006481Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:32.905634995Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:33.21392971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:33.21425704Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:33.7952439Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:33.795977133Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:33.816488178Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:33.81758312Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:33.819812936Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:34.132391935Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:34.132644201Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:35.309213854Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:35.576867303Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:35.577323897Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:36.601909166Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:36.602318171Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:36.921389184Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:37.161015238Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:37.161464994Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:39.222770971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:39.223197048Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:40.701374505Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:40.701630451Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:35:40.859448855Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:41.245234076Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:41.245559678Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:41.491166647Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:41.514282908Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:41.514653021Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:41.565719416Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:41.566019734Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:41.734105605Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:42.546993086Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:42.547452438Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:43.858851575Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:44.000215003Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:44.273027683Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:45.113506612Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:45.113847544Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:45.358073136Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:45.358900016Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:45.733183961Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:45.733875381Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:45.924024331Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:45.924632333Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:46.374651331Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:46.766047297Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:46.76649574Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:46.986538023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:47.073739656Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:47.074068754Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:48.259026944Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:48.564482972Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:48.651029275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:48.651621313Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:49.200838005Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:49.55538087Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:49.555604806Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:49.682008266Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:35:50.859425434Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:51.704035821Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:51.704817249Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:51.897995707Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:51.898602042Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:51.982925147Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:52.090840188Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:52.091294933Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:52.57782689Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:52.691010393Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:53.05677006Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:53.057365581Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:53.331934205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:53.896587391Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:54.981033907Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:54.981213531Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:55.3075527Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:55.307991724Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:55.412026729Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:55.412178163Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:55.782521687Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:55.78302375Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:56.402774965Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:58.262561503Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:58.851385801Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:59.294005482Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:59.294331791Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:59.711764852Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:35:59.712015579Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:00.036984916Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:00.03711797Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:36:00.859419651Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:00.93038418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:01.070608727Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:01.071127731Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:01.756479058Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:02.458637036Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:02.458850067Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:02.684763418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:02.685492137Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:03.699231862Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:04.678560332Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:04.722604898Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:07.177430202Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:07.210370297Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:07.233142344Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:07.233378147Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:07.26601309Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:07.266753777Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:08.01325974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:08.521388834Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:09.392296501Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:09.392628938Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:09.691549248Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:09.692113001Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:10.017581137Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:10.432833789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:36:10.864292575Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:12.441572638Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:12.442175703Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:12.516109944Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:14.848969236Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:16.723829293Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:16.724287067Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:17.173008Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:17.454971145Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:17.455293489Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:18.192527112Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:19.410165828Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:19.694146586Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:19.694765721Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:36:20.859441931Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:21.213654087Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:21.213986984Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:21.922509495Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:21.922686393Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:22.764361006Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:23.11692754Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:23.117193602Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:25.697534015Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:25.737392117Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:25.737585315Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:26.272290924Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:26.272605115Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:27.311175375Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:27.311470587Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:28.178173808Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:28.178835411Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:28.715799813Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:28.71631881Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:29.096810362Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:36:30.859431447Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:31.652486784Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:31.653069276Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:31.795009317Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:31.795269674Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:31.985884617Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:31.986278468Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:32.440737761Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:32.441091377Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:32.635492986Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:32.635994128Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:32.905537512Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:33.213976794Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:33.214300381Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:33.795440119Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:33.796232741Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:33.818589362Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:33.819699567Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:33.822012641Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:34.133302942Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:34.133623819Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:35.310308812Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:35.577412804Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:35.577907957Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:36.59901Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:36.59938546Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:36.945546785Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:37.169920862Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:37.170370073Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:39.266429196Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:39.268985345Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:40.700849918Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:40.701108705Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:36:40.859429584Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:41.246655168Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:41.246974058Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:41.487994992Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:41.52854617Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:41.529156378Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:41.556527252Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:41.556884735Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:41.719227344Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:42.569864621Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:42.570291867Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:43.950622459Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:43.998512309Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:44.337356161Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:44.703468589Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:44.704047657Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:45.358098728Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:45.358839717Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:45.714017415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:45.714651402Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:45.938368526Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:45.939005123Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:46.374978719Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:46.602712287Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:46.603006214Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:46.982765203Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:47.06319372Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:47.063528147Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:48.264414222Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:48.554278612Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:48.643130544Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:48.644002217Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:49.20138825Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:49.554694888Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:49.554934118Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:49.649153726Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:36:50.859441505Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:51.708015477Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:51.708723035Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:51.899061019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:51.899757564Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:51.985389531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:52.076598966Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:52.07703106Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:52.587447527Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:52.697211428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:52.998933484Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:52.999386015Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:53.331940754Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:53.898043113Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:54.972966067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:54.973172828Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:55.295869846Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:55.296320417Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:55.484714402Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:55.484916771Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:55.639635478Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:55.640073428Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:56.40255756Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:58.276425163Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:58.853896431Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:59.098355223Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:59.098710823Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:59.712210758Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:59.712538842Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:59.889567222Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:36:59.889754725Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:37:00.859433613Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:00.93127168Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:01.017939453Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:01.018444566Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:02.003256629Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:02.373458687Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:02.373639747Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:02.672831186Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:02.673543665Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:03.698534155Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:04.674074157Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:04.720000797Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:07.174419897Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:07.204804197Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:07.23170376Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:07.232011108Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:07.256382008Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:07.257090705Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:08.02120857Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:08.11443767Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:09.378266964Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:09.378590425Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:09.691236166Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:09.691725858Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:10.01049349Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:10.431001831Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:37:10.872123933Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:12.43303314Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:12.433628828Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:12.599704036Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:14.841834778Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:16.841751039Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:16.842193672Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:17.073323268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:17.467810755Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:17.468134612Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:18.166138723Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:19.408042892Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:19.695977779Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:19.696441333Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:37:20.859505433Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:21.209344223Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:21.209690571Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:21.912468369Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:21.912645549Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:22.75802634Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:23.116681586Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:23.116929994Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:25.697689313Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:25.726973871Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:25.727206021Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:26.272128289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:26.272434961Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:27.383189074Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:27.383482814Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:28.195899334Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:28.196491389Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:28.558366569Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:28.559133926Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:28.931832199Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:37:30.859445992Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:31.640326546Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:31.640898782Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:31.769217392Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:31.769452529Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:32.275739328Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:32.276106003Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:32.444780427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:32.445117776Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:32.593120379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:32.593558769Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:32.905604504Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:33.214399579Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:33.214759218Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:33.795338496Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:33.796172947Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:33.818402699Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:33.82011785Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:33.823082456Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:34.133492106Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:34.133825125Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:35.319134943Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:35.576782679Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:35.577211877Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:36.620890933Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:36.621349742Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:36.920995011Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:37.161293339Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:37.16176689Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:39.122705592Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:39.123374985Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:40.696933986Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:40.697227589Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:37:40.85948734Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:41.245414759Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:41.245775381Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:41.487682946Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:41.519706777Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:41.520058748Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:41.555387317Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:41.55568697Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:41.722175603Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:42.548994444Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:42.549429179Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:43.859284069Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:43.998456277Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:44.337978805Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:45.132947474Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:45.133259067Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:45.358093567Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:45.35882109Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:45.724771337Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:45.725406926Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:45.918058086Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:45.918690873Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:46.374823378Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:46.603437849Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:46.603757867Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:46.986039436Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:47.0701833Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:47.070571134Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:48.255548873Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:48.553173481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:48.65224858Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:48.652873845Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:49.201203154Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:49.555350117Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:49.555620053Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:49.652848112Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:37:50.859428834Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:51.709259629Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:51.709960039Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:51.889044465Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:51.889585488Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:51.9854934Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:52.082290452Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:52.082748217Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:52.577071171Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:52.691085638Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:52.99886703Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:52.999309434Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:53.329490468Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:53.896376446Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:54.972433539Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:54.97264211Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:55.33281529Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:55.333396845Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:55.450318302Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:55.450475985Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:55.679048639Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:55.679485621Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:56.401955612Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:58.257979708Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:58.857925891Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:59.231986713Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:59.23265541Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:59.718715756Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:37:59.719028249Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:00.059007668Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:00.059170302Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:00.853509698Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:00.854003029Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:38:00.859410269Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:00.930339717Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:01.509539924Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:02.380770923Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:02.38092768Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:02.676925655Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:02.677874008Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:03.697931715Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:04.67804847Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:04.7198677Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:07.171763011Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:07.201635279Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:07.220642164Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:07.220858546Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:07.269177014Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:07.269942829Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:08.017560447Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:08.55358772Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:09.379150578Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:09.379555875Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:09.735824961Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:09.736827654Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:10.013013992Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:10.437438259Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:38:10.859432942Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:12.444749601Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:12.445398166Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:12.520682764Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:14.84993972Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:16.841365167Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:16.842024925Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:17.041368479Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:17.414321006Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:17.414608026Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:18.185926582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:19.407068298Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:19.69814379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:19.698654887Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:38:20.859433644Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:21.218811243Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:21.219152208Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:21.921407958Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:21.921556268Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:22.758250418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:23.128082167Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:23.128335313Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:25.697901571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:25.74402894Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:25.74428095Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:26.275763349Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:26.27603812Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:27.378512731Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:27.378817446Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:28.173344428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:28.17788885Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:28.527426102Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:28.5279286Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:29.036540446Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:38:30.859447011Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:31.628790955Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:31.629278257Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:31.769359Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:31.769595041Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:32.283190173Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:32.283628998Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:32.34496865Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:32.345338236Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:32.592304531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:32.592770908Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:32.905775406Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:33.213816843Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:33.214121822Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:33.795289913Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:33.796048045Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:33.815957478Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:33.817078301Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:33.819328458Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:34.132461797Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:34.132749866Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:35.309953101Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:35.576968606Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:35.577406395Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:36.617572468Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:36.617972183Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:36.920257458Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:37.160878879Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:37.161324608Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:39.291502277Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:39.292031926Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:40.685786523Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:40.686061306Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:38:40.859433571Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:41.255599741Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:41.255952843Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:41.487649612Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:41.566002597Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:41.566445065Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:41.593191006Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:41.593484025Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:41.774823543Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:42.547245937Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:42.547704411Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:43.856248172Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:43.998032799Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:44.339282484Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:44.56821764Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:44.56850184Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:45.358091399Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:45.358875043Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:45.72685369Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:45.727541222Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:45.930320592Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:45.930957432Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:46.375646459Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:46.601631542Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:46.602104592Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:46.981123863Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:47.075343604Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:47.075642996Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:48.258589283Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:48.553061406Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:48.650283818Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:48.650932918Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:49.201585414Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:49.494473712Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:49.494987853Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:49.63898033Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:38:50.859439239Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:51.710935583Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:51.711909866Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:51.913813913Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:51.914483161Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:51.982418563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:52.086777523Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:52.087223178Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:52.577715459Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:52.698894965Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:52.999018213Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:52.999550274Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:53.322110133Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:53.89721195Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:54.976848969Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:54.977041536Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:55.342713538Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:55.343167298Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:55.47049682Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:55.470631812Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:55.654195999Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:55.654588864Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:56.405738865Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:58.270083249Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:58.851537442Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:59.285493427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:59.285770372Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:59.71201776Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:38:59.712276512Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:00.045075784Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:00.04526668Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:39:00.859551195Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:00.862787728Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:00.863271636Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:00.931300216Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:01.590449267Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:02.358736009Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:02.358885465Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:02.677494136Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:02.678278476Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:03.697587493Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:04.694164523Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:04.720321553Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:07.207582324Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:07.253474095Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:07.265859738Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:07.266907923Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:07.275726763Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:07.27596022Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:08.010842147Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:08.936217499Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:09.516005998Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:09.516415786Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:09.726257963Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:09.742544959Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:10.027720058Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:10.433055079Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:39:10.859445995Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:12.438889372Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:12.439469928Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:12.518364689Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:14.855965931Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:16.715124949Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:16.715551773Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:17.071708462Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:17.487090018Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:17.487476618Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:18.18929586Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:19.506120073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:19.690636781Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:19.691179062Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:39:20.8594509Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:21.210089563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:21.210445593Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:21.919911334Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:21.920459522Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:22.758186247Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:23.116499109Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:23.116803556Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:25.697979523Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:25.743781817Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:25.744018395Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:26.271693499Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:26.271955809Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:27.311435834Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:27.311725148Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:28.158211767Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:28.158783592Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:28.537274499Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:28.537825824Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:28.936794648Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:39:30.859416107Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:31.635554557Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:31.63640634Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:31.770321417Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:31.770563584Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:32.280936546Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:32.281364739Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:32.348875619Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:32.349207Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:32.59231445Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:32.592799803Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:32.905494939Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:33.214509507Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:33.214836725Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:33.795426345Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:33.797425192Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:33.817225223Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:33.818326465Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:33.820491997Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:34.132378406Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:34.132627711Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:35.310183741Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:35.586680925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:35.587143555Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:36.694969876Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:36.695449212Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:36.996109365Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:37.167286017Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:37.167883895Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:39.154298242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:39.155862163Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:40.687009304Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:40.687227616Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:39:40.859448236Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:41.259576748Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:41.259923915Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:41.487639295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:41.511399198Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:41.511831758Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:41.569974235Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:41.570285842Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:41.706766428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:42.549825257Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:42.55029881Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:43.859087513Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:44.00065012Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:44.272579524Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:45.150486915Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:45.150864285Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:45.358048367Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:45.358817095Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:45.734471582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:45.73515044Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:45.954835165Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:45.955446502Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:46.376048905Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:46.602414322Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:46.602762629Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:46.98233214Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:47.077188399Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:47.077558551Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:48.255620554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:48.557262391Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:48.653952954Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:48.654527024Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:49.201392233Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:49.493212704Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:49.493464483Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:49.654708343Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:39:50.859430046Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:51.708850343Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:51.709560163Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:51.894802412Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:51.895322845Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:51.983682984Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:52.066363572Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:52.066761988Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:52.576991721Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:52.691510537Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:52.99983947Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:53.000311181Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:53.354366416Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:53.898618187Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:54.981966466Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:54.982176739Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:55.334495819Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:55.33494823Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:55.433259058Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:55.433465055Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:55.850884326Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:55.851380256Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:56.401888377Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:58.335382335Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:58.864254571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:59.291169982Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:59.291480426Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:59.724603922Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:39:59.724978206Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:00.050805672Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:00.050961944Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:40:00.859425249Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:00.930381336Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:01.046910806Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:01.047387213Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:01.67651644Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:02.688076703Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:02.688272291Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:03.83933406Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:03.866975327Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:03.875261894Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:04.691895603Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:04.726777309Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:07.171034768Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:07.200438671Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:07.227241898Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:07.227508684Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:07.274572548Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:07.275589075Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:08.040194705Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:08.485345853Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:09.34644085Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:09.346998504Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:09.717752312Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:09.718338876Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:10.012378444Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:10.448975848Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:40:10.863864063Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:12.488162581Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:12.48896814Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:12.598248595Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:14.842083336Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:17.032247497Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:17.040483172Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:17.078033402Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:17.55254337Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:17.557362911Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:18.607793133Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:19.414100667Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:19.773172727Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:19.773918266Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:40:20.859482645Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:21.935441753Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:21.935801086Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:22.006789073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:22.007237955Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:22.804456595Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:23.116919041Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:23.117181418Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:25.697697842Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:25.744590103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:25.744858817Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:26.272655043Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:26.273022204Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:27.382926067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:27.383915458Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:28.160425554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:28.16102293Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:28.759900926Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:28.760422224Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:28.944126604Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:40:30.859462574Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:31.667465917Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:31.66822587Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:31.77305119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:31.773402607Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:31.978718413Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:31.979118674Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:32.352730641Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:32.353082304Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:32.593453707Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:32.593942627Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:32.905566324Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:33.214699415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:33.215089413Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:33.795768984Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:33.796653146Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:33.827269032Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:33.829580895Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:33.833039267Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:34.165961732Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:34.166217142Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:35.308938684Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:35.576938386Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:35.577411755Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:36.63019097Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:36.630599138Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:36.966257975Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:37.161071493Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:37.161568342Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:39.268182666Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:39.268649591Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:40.702325758Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:40.702566672Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:40:40.85941163Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:41.247127301Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:41.247392268Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:41.488169063Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:41.526726615Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:41.527105755Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:41.559936269Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:41.560248493Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:41.720801322Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:42.547031415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:42.547478076Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:43.873037434Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:43.999999121Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:44.339301269Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:45.129029854Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:45.129362631Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:45.358186896Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:45.358932688Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:45.716918597Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:45.717713413Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:45.930209348Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:45.930825954Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:46.374761956Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:46.598377859Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:46.59872073Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:46.985568119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:47.060270607Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:47.060589921Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:48.255984893Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:48.579306291Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:48.753028232Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:48.753888309Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:49.212166517Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:49.560187943Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:49.560476659Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:49.638338931Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:40:50.859467918Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:51.746855632Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:51.747974735Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:51.903751035Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:51.904254259Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:51.987204335Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:52.071107295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:52.071611083Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:52.586054869Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:52.692065588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:52.998827205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:52.999272339Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:53.322800042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:53.901439928Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:54.978725611Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:54.978925479Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:55.297898656Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:55.298348871Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:55.471414643Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:55.471580109Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:55.769045492Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:55.769503141Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:56.409334399Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:58.290162385Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:58.855582777Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:59.123480147Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:59.123850096Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:59.711528176Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:59.71183625Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:59.870603899Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:40:59.870772981Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:41:00.859433934Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:00.931317578Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:01.089388424Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:01.089981604Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:01.635451211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:02.348100563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:02.34824839Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:02.680133412Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:02.680828753Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:03.699097792Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:04.680735216Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:04.722537585Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:07.188462883Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:07.21877852Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:07.239302967Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:07.239597805Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:07.262137415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:07.262880853Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:08.013503806Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:08.841438525Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:09.370118682Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:09.370441864Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:09.694278161Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:09.694949579Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:10.011635803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:10.432157342Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:41:10.859451472Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:12.440157282Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:12.440785246Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:12.595248248Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:14.840653774Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:16.842560554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:16.843155323Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:17.070716958Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:17.460523569Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:17.460818942Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:18.158099165Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:19.413641242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:20.004309282Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:20.004880628Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:41:20.859448119Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:21.222809582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:21.223235065Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:21.90604903Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:21.906195467Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:22.758255559Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:23.116813287Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:23.117094612Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:25.698263956Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:25.744570889Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:25.744890154Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:26.272520166Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:26.272805954Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:27.378693531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:27.378954129Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:28.160207014Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:28.160829964Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:28.783333083Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:28.783913572Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:28.934047445Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:41:30.859424085Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:31.634077311Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:31.634640975Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:31.76957901Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:31.769919558Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:31.963783396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:31.964205688Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:32.353854127Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:32.354253091Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:32.573000528Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:32.573452108Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:32.90561496Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:33.214588134Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:33.21492844Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:33.795274194Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:33.795999737Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:33.815654353Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:33.81678385Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:33.818900396Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:34.157255202Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:34.157628796Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:35.309976729Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:35.576950555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:35.577452284Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:36.61180359Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:36.612297277Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:36.929056076Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:37.173233658Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:37.173687462Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:39.421759914Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:39.422382464Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:40.688820567Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:40.689152397Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:41:40.859438038Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:41.26771579Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:41.268030077Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:41.486878757Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:41.522834207Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:41.523216468Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:41.564435477Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:41.56477668Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:41.713834978Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:42.210183961Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:42.210568506Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:43.857730555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:43.999549427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:44.272615321Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:45.158093829Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:45.158387368Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:45.358095323Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:45.358820704Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:45.713782939Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:45.714433202Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:45.9303543Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:45.930984202Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:46.373966492Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:46.650854577Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:46.651258817Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:46.983928179Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:47.074135383Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:47.074524408Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:48.254915083Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:48.554315882Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:48.66617844Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:48.666777704Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:49.200267005Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:49.555473761Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:49.555765242Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:49.650049599Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:41:50.859459842Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:51.709306531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:51.710047056Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:51.902202238Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:51.902743256Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:51.984801369Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:52.069144023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:52.069544551Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:52.577504953Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:52.692739644Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:53.00096315Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:53.001389839Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:53.32420664Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:53.897611596Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:54.978194552Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:54.978399939Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:55.305850209Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:55.306334304Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:55.416274978Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:55.416425258Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:55.768513858Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:55.76898625Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:56.402802875Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:58.290977525Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:58.853227156Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:59.292703964Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:59.293025184Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:59.712143893Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:41:59.712411881Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:00.056320561Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:00.05648141Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:42:00.859448249Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:00.930619561Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:01.036418179Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:01.03691045Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:02.028940984Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:02.367506427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:02.367699084Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:02.790833495Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:02.791532087Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:03.699830198Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:04.682963996Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:04.719851248Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:07.175676011Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:07.206804725Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:07.225388071Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:07.22559929Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:07.269914539Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:07.270747255Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:08.011617032Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:08.493485175Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:08.895605563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:08.896021305Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:09.701225664Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:09.70231857Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:10.060730011Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:10.431456982Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:42:10.859429434Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:12.431436386Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:12.432068419Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:12.595968236Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:14.841507712Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:16.840769234Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:16.841217973Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:17.076977718Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:17.424254778Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:17.424584244Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:18.14761456Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:19.408218298Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:19.721758867Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:19.723104002Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:42:20.859431174Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:21.214734164Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:21.215195979Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:21.932139974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:21.932378153Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:22.773454265Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:23.117735758Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:23.118026088Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:25.697515751Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:25.7483114Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:25.748587422Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:26.272087706Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:26.272379743Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:27.388135066Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:27.388410213Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:28.201340847Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:28.202300983Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:28.772396634Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:28.773146674Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:28.969779469Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:42:30.859437967Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:31.659971498Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:31.660514016Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:31.769853797Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:31.770055986Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:31.965768132Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:31.96618653Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:32.453400094Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:32.453793579Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:32.593195599Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:32.593679543Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:32.905525866Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:33.224687687Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:33.225021827Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:33.795446424Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:33.796215264Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:33.818305386Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:33.819421122Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:33.821729945Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:34.132501611Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:34.132808533Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:35.308868839Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:35.588865842Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:35.589323895Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:36.615052826Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:36.615509429Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:36.921024108Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:37.161847436Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:37.162716736Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:39.241543732Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:39.241948598Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:40.685971979Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:40.686226528Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:42:40.859441066Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:41.254992805Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:41.255280156Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:41.487194283Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:41.510883384Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:41.511249877Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:41.555816639Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:41.556109743Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:41.735136395Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:42.539366422Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:42.539774007Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:43.862747912Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:43.997504172Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:44.28115511Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:45.131319491Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:45.131643514Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:45.358116995Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:45.358983226Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:45.734096238Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:45.734803976Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:45.921135565Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:45.921767955Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:46.379166809Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:46.628782827Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:46.629228624Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:46.982112696Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:47.226524015Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:47.226867417Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:48.254112729Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:48.554973096Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:48.651824506Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:48.652313815Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:49.200709805Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:49.554967485Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:49.555198804Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:49.637232554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:42:50.859437736Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:51.703364671Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:51.704103268Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:51.901956617Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:51.902622737Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:51.984060422Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:52.081563781Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:52.081964379Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:52.577132874Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:52.692356264Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:52.998659103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:52.99913684Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:53.301358956Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:53.896304239Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:54.972497729Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:54.972734663Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:55.296304048Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:55.296783591Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:55.470083547Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:55.470254746Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:55.784763929Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:55.785212777Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:56.403284579Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:58.279269373Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:58.855378077Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:59.28643423Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:59.286749205Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:59.712244126Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:42:59.712582877Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:00.055891805Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:00.056028003Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:43:00.859441084Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:00.930625367Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:01.149634046Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:01.150138671Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:01.865241888Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:02.368977218Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:02.369155097Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:02.665973932Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:02.666737132Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:03.698185387Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:04.676358501Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:04.720525161Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:07.183980187Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:07.218456536Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:07.238297198Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:07.238543895Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:07.255396026Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:07.256142259Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:08.029454564Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:08.261642952Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:09.441079213Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:09.441461426Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:09.684629795Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:09.68543665Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:10.018226976Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:10.431615933Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:43:10.859427024Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:12.440187698Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:12.440846898Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:12.598377898Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:14.840743047Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:16.608314552Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:16.608933937Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:16.726343927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:16.726830309Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:17.072365767Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:18.177127323Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:19.404962558Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:19.695038702Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:19.695569382Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:43:20.85951565Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:21.208829265Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:21.209130472Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:21.938886788Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:21.939053087Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:22.77268477Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:23.117722967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:23.118125936Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:25.697578268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:25.744003923Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:25.744243842Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:26.272887827Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:26.273154924Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:27.311029431Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:27.3117897Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:28.16058852Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:28.161480631Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:28.748104609Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:28.748624092Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:28.934080024Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:43:30.859408561Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:31.621917278Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:31.622480333Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:31.77015841Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:31.770404176Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:32.290807348Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:32.291152608Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:32.445756312Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:32.44612406Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:32.602659379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:32.603169465Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:32.905559578Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:33.214164069Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:33.214478396Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:33.795305644Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:33.796204186Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:33.816856894Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:33.817997327Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:33.820195156Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:34.13231525Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:34.13259932Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:35.311245738Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:35.577826953Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:35.578285108Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:36.654074826Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:36.654456514Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:36.919780883Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:37.161850866Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:37.162250528Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:39.165383097Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:39.166182329Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:40.687536773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:40.687802324Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:43:40.859427258Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:41.260569712Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:41.260892404Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:41.489540209Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:41.512747336Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:41.513066414Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:41.557806275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:41.558073531Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:41.71901593Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:42.546654933Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:42.547099465Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:43.857704111Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:43.99799683Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:44.338032599Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:44.562549742Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:44.562884434Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:45.358070986Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:45.358831022Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:45.727493452Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:45.728197538Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:45.932838142Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:45.933398335Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:46.378776014Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:46.592285337Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:46.592618225Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:46.985411192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:47.212066504Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:47.212378023Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:48.256931839Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:48.521347246Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:48.521866334Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:48.554018334Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:49.200341117Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:49.556388292Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:49.556627448Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:49.662401976Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:43:50.859454243Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:51.710331598Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:51.711012693Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:51.90038412Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:51.901039421Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:51.98449052Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:52.072678686Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:52.073087029Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:52.577457298Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:52.691679134Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:52.998802489Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:52.999237494Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:53.300609342Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:53.901182889Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:54.973322043Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:54.973514009Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:55.296041422Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:55.29649618Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:55.472353502Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:55.47253344Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:55.744864383Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:55.745284228Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:56.402235317Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:58.286284536Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:58.865226774Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:59.292925177Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:59.293282437Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:59.713041306Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:59.713344111Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:59.951304352Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:43:59.951475412Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:44:00.859458586Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:00.930780568Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:01.065600237Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:01.066039964Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:02.127849798Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:02.377117618Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:02.377274337Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:02.677215856Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:02.677981467Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:03.701396139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:04.679288433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:04.722847415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:07.190389535Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:07.233099476Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:07.246136405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:07.246931745Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:07.252498723Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:07.252729822Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:08.022329191Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:09.598234691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:09.598616452Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:09.696853384Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:09.697408884Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:10.044996437Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:10.229043027Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:10.473594265Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:44:10.8594304Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:12.430854121Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:12.431478601Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:12.515180229Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:14.842470365Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:16.72413139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:16.72456568Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:17.044313716Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:17.18604468Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:17.186422318Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:18.355617296Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:19.410100821Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:20.010916788Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:20.011399043Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:44:20.859837956Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:21.236594846Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:21.237436743Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:21.903157515Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:21.903511809Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:22.757989767Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:23.115919826Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:23.116140319Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:25.698318083Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:25.747636146Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:25.747928034Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:26.272239955Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:26.272538864Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:27.38160309Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:27.381869073Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:28.178532305Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:28.179232414Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:28.551550192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:28.552402577Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:28.944020052Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:44:30.859439103Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:31.659908321Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:31.660471637Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:31.76963008Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:31.769875604Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:32.288589209Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:32.288977309Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:32.45089427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:32.45135547Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:32.592241435Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:32.592714108Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:32.90553071Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:33.215220305Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:33.215551093Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:33.795399278Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:33.796125684Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:33.816129267Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:33.817239348Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:33.81938522Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:34.137998124Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:34.138352072Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:35.309586243Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:35.57727653Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:35.577772571Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:36.750522002Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:36.750998342Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:36.92752384Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:37.170260931Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:37.170724004Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:39.133534758Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:39.133919202Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:40.718765802Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:40.719039038Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:44:40.859749038Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:41.27072708Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:41.271127043Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:41.430892907Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:41.431289568Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:41.487747974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:41.558389915Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:41.558896615Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:41.716557642Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:42.22318734Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:42.223702279Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:43.860270363Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:43.997854726Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:44.338624967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:45.118659007Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:45.118937998Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:45.358278258Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:45.359108322Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:45.733182156Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:45.73387782Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:45.922720336Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:45.92330993Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:46.374569226Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:46.605031899Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:46.605376229Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:46.981100796Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:47.077768588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:47.078166103Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:48.272384084Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:48.520078063Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:48.520525973Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:48.552115105Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:49.200511488Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:49.503416893Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:49.503621394Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:49.635373248Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:44:50.859442281Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:51.711444157Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:51.712124167Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:51.894242593Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:51.894808119Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:52.081704275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:52.086996652Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:52.087401096Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:52.577557325Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:52.693645691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:52.998808099Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:52.999284994Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:53.300593314Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:53.8979433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:54.981847472Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:54.982075258Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:55.33254045Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:55.333090159Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:55.413022659Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:55.413176827Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:55.672436494Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:55.672896757Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:56.40515994Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:58.282881595Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:58.854050652Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:59.304247133Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:59.304567625Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:59.713573877Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:44:59.713862518Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:00.052948228Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:00.053111522Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:45:00.859440777Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:00.930279377Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:01.024993934Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:01.025425712Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:02.469053666Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:02.47051652Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:02.907566737Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:02.909529472Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:03.11193863Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:03.69848598Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:04.680207716Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:04.752194657Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:07.177695959Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:07.212799715Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:07.263249243Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:07.263613226Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:07.29158196Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:07.292364009Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:08.011864722Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:09.622098049Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:09.622504458Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:09.704816466Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:09.705406758Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:09.892928276Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:10.017996369Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:10.430586919Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:45:10.859439557Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:12.4531872Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:12.454761855Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:12.526207139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:14.846392295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:16.874757507Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:16.875195954Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:17.171036343Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:17.502302615Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:17.502710333Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:18.261753485Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:19.408020499Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:19.691489494Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:19.691987551Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:45:20.859429903Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:21.23834898Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:21.23884318Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:21.911136621Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:21.911287361Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:22.773381681Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:23.117210569Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:23.11750743Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:25.69913463Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:25.745990382Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:25.746218568Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:26.271898507Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:26.272170003Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:27.38168565Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:27.381957163Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:28.17542883Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:28.176085812Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:28.600396309Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:28.600949126Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:28.939281879Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:45:30.859446713Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:31.627838029Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:31.628550201Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:31.771034688Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:31.771266746Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:32.286762103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:32.287163416Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:32.469703018Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:32.470149625Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:32.592584112Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:32.593042149Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:32.905732562Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:33.213634476Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:33.213976136Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:33.79581717Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:33.796858558Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:33.81977992Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:33.820960485Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:33.823227811Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:34.133537745Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:34.133800473Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:35.309307242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:35.577469019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:35.578095695Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:36.654224761Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:36.654755219Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:36.922111971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:37.171420958Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:37.171851464Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:39.191405882Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:39.191798555Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:40.887115786Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:40.887432905Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:45:40.888129687Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:41.133123956Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:41.135269869Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:41.486563955Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:41.557495752Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:41.557974501Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:41.575178846Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:41.575437156Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:41.717927667Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:42.556636947Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:42.557108592Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:43.951106694Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:43.998902539Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:44.338126412Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:45.115189936Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:45.11549868Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:45.358256474Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:45.359107895Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:45.730545065Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:45.731455027Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:45.926131476Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:45.926767357Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:46.37609645Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:46.773446131Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:46.773889657Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:46.987440394Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:47.062007174Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:47.062324349Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:48.272759381Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:48.554050071Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:48.617069994Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:48.617643162Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:49.204687702Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:49.516452252Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:49.517219307Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:49.655016243Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:45:50.859433554Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:51.74347515Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:51.744460048Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:51.91706332Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:51.917721565Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:51.987798739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:52.066267455Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:52.066622632Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:52.577541822Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:52.691020443Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:52.999031047Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:52.999489526Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:53.300598783Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:53.896930721Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:54.9823732Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:54.982589161Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:55.338003362Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:55.338436144Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:55.413800022Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:55.413957017Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:55.65206004Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:55.652475193Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:56.402948171Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:58.297444764Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:58.852049465Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:59.28907634Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:59.289334325Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:59.716329444Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:59.716627668Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:59.887832673Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:45:59.887995076Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:46:00.859424075Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:00.932057826Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:01.114593948Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:01.115065315Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:02.232048412Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:02.377769847Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:02.377914523Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:02.672148378Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:02.672827872Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:03.698106886Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:04.677792699Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:04.733878899Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:07.194818642Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:07.22616735Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:07.245395109Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:07.245615726Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:07.327176209Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:07.328082343Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:08.010522587Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:09.184087814Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:09.562480372Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:09.563109634Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:09.718041067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:09.71922034Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:10.041314643Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:10.434737971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:46:10.859515232Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:12.44552655Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:12.446164746Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:12.520813828Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:14.845490671Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:16.842373638Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:16.842824113Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:17.075883517Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:17.461745374Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:17.462102907Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:18.144054759Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:19.420771458Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:19.988890833Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:19.98943092Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:46:20.859440642Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:21.210416601Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:21.210782689Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:21.933718525Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:21.933901473Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:22.764850568Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:23.125190642Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:23.125653828Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:25.697033103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:25.746270969Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:25.746518336Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:26.344157401Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:26.344428342Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:27.311722406Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:27.312021049Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:28.163254632Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:28.163926211Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:28.729343965Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:28.730009025Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:28.981000186Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:46:30.859436818Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:31.644491298Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:31.645055719Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:31.769364377Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:31.769597707Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:32.214980725Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:32.215379569Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:32.401271703Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:32.401626717Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:32.593038852Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:32.593473385Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:32.905510788Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:33.2261022Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:33.226501063Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:33.795459371Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:33.796191781Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:33.817908857Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:33.819427584Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:33.822301319Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:34.132782951Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:34.133044844Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:35.309523057Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:35.577225306Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:35.577645612Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:36.616683342Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:36.617161015Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:36.921912898Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:37.161566658Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:37.161995698Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:39.115923171Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:39.116262153Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:40.707580772Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:40.710182403Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:46:40.869031912Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:41.256253908Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:41.256583126Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:41.487465221Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:41.520080889Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:41.520401016Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:41.558113827Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:41.558414688Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:41.719119182Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:42.53699912Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:42.537426684Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:43.870479642Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:43.998330397Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:44.338631039Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:44.974722363Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:44.975049214Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:45.358105173Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:45.358834523Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:45.728125243Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:45.728739101Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:45.925592076Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:45.926389688Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:46.375337683Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:46.602353303Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:46.602724853Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:46.985142237Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:47.059420689Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:47.059772761Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:48.281314488Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:48.561593458Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:48.659585103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:48.662222322Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:49.202615254Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:49.555377275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:49.5556085Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:49.661967795Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:46:50.859431026Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:51.709492447Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:51.710188666Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:51.906767903Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:51.907381952Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:51.983213228Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:52.071864898Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:52.072244361Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:52.578101023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:52.6994546Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:53.010790379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:53.011272729Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:53.300915582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:53.897383309Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:54.98041366Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:54.980598032Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:55.332513174Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:55.332968138Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:55.429209781Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:55.429349721Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:55.786556403Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:55.787014213Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:56.402475344Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:58.310809278Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:58.853429901Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:59.10560115Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:59.106098922Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:59.711599786Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:59.711909165Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:59.886328053Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:46:59.886492917Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:47:00.859438393Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:00.930307812Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:01.026098804Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:01.026608025Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:01.454698159Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:02.346554942Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:02.346726689Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:02.669919169Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:02.67059059Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:03.698303783Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:04.675875166Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:04.720189077Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:07.18749586Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:07.221375813Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:07.251326819Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:07.251703856Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:07.285133739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:07.286272728Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:08.025643399Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:08.025629318Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:09.420570307Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:09.421194687Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:09.704905271Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:09.706192459Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:10.023213683Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:10.432706748Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:47:10.859446589Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:12.439008409Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:12.439580638Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:12.596525518Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:14.846267574Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:16.840979602Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:16.841453876Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:17.041933871Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:17.432125719Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:17.432486031Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:18.222692759Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:19.407489917Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:19.685640341Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:19.686180734Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:47:20.859432888Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:21.209369125Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:21.209682886Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:21.927316189Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:21.927501333Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:22.774087042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:23.116706937Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:23.116930497Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:25.697777968Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:25.754867817Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:25.755086793Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:26.271947143Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:26.27225598Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:27.378223447Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:27.378488438Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:28.21333032Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:28.214179165Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:28.686894747Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:28.687333007Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:28.931439207Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:47:30.871783847Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:31.773094105Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:31.773358869Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:31.961900004Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:31.962303252Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:32.365631965Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:32.366007364Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:32.523795068Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:32.524326606Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:32.602965353Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:32.603414986Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:32.905731786Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:33.213648774Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:33.214014029Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:33.795236944Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:33.795942138Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:33.815499801Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:33.816655971Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:33.818874485Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:34.150026447Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:34.15032193Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:35.308704854Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:35.576646057Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:35.577109558Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:36.632584723Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:36.63322Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:36.922060879Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:37.162114246Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:37.162866412Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:39.089728544Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:39.090216551Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:40.719187851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:40.719504188Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:47:40.863364732Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:41.263883897Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:41.264184904Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:41.487984013Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:41.512774067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:41.513171557Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:41.557214741Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:41.557470784Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:41.716489761Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:42.55350579Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:42.553937687Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:43.857041693Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:43.997588174Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:44.339409609Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:45.118985134Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:45.11923857Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:45.35809167Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:45.358857881Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:45.767532711Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:45.768421223Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:45.939597869Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:45.940303117Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:46.377743589Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:46.60863192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:46.608998457Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:46.985994787Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:47.059711275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:47.060025324Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:48.275151921Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:48.553285497Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:48.652305728Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:48.653142829Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:49.200553765Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:49.554560058Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:49.554801689Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:49.644879768Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:47:50.859446902Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:51.708631791Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:51.709340293Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:51.920390165Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:51.921066341Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:51.982194876Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:52.063232785Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:52.063641597Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:52.576247754Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:52.690991778Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:52.9987007Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:52.999145913Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:53.301036291Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:53.896938111Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:54.977458707Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:54.977722596Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:55.333359717Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:55.333823683Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:55.413118392Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:55.413265408Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:55.746681289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:55.74714961Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:56.405418593Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:58.254552073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:58.856574985Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:59.089821101Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:59.090236449Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:59.711371361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:59.711650986Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:59.887237371Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:47:59.887393025Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:48:00.859422968Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:00.931096962Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:01.17232626Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:01.172856823Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:02.122479638Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:02.346218413Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:02.346349448Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:02.673425968Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:02.674121106Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:03.698085616Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:04.68236209Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:04.721102696Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:07.180455372Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:07.211776335Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:07.231320905Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:07.231527596Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:07.252369759Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:07.253168525Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:08.011943728Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:08.369819851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:09.47356528Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:09.474069632Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:09.714263686Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:09.720370745Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:10.115404624Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:10.432948091Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:48:10.859438831Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:12.44131667Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:12.441909035Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:12.515240708Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:14.84276419Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:16.748214433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:16.748831033Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:17.077577595Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:17.459537942Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:17.459820349Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:18.203823235Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:19.407185131Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:19.693353573Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:19.693902855Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:48:20.859430773Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:21.207502584Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:21.20784594Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:21.867382051Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:21.86754451Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:22.77336631Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:23.115744957Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:23.116022109Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:25.697377581Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:25.745142439Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:25.745469206Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:26.272237839Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:26.272521058Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:27.378234852Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:27.378513437Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:28.162638136Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:28.16324627Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:28.945950079Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:29.267971244Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:29.268466798Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:48:30.859439743Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:31.770007917Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:31.770259417Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:31.964401102Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:31.964899385Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:32.354752575Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:32.355129059Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:32.461364619Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:32.461941199Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:32.592992164Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:32.593436511Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:32.90554945Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:33.213954948Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:33.214269546Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:33.7952501Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:33.795972131Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:33.815345441Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:33.816456057Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:33.818614169Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:34.150429014Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:34.150684863Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:35.310163217Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:35.577101406Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:35.577556339Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:36.606205693Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:36.60655434Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:36.964740894Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:37.161113076Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:37.161575371Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:39.181226483Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:39.181523781Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:40.716138835Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:40.716408203Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:48:40.859452508Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:41.207757994Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:41.208048799Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:41.508426Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:41.59645457Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:41.596800101Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:41.695359672Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:41.695850919Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:41.80234608Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:42.563030522Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:42.563443669Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:43.872185053Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:43.998036691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:44.275948334Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:44.549573129Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:44.549901305Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:45.358110245Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:45.358895921Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:45.722685977Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:45.723526868Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:45.923065031Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:45.92369171Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:46.375732358Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:46.600960978Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:46.601337158Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:46.985101066Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:47.075526995Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:47.075879934Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:48.274699912Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:48.553818126Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:48.651755997Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:48.652361537Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:49.201676097Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:49.555310437Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:49.555535687Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:49.642824049Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:48:50.859434959Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:51.70301426Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:51.704183768Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:51.893291616Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:51.893901531Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:51.984113874Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:52.090328395Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:52.090877044Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:52.576415745Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:52.692373012Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:52.999224762Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:52.999804921Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:53.327154985Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:53.897311451Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:54.980834788Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:54.981053847Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:55.295958315Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:55.296421616Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:55.430411132Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:55.430570518Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:55.753380664Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:55.753803465Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:56.404007441Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:58.25472128Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:58.856732619Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:59.2900817Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:59.290386843Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:59.713104866Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:59.713379298Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:59.887547135Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:48:59.887711257Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:49:00.859442015Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:00.9308938Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:01.128467811Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:01.128951112Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:02.194272683Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:02.348544104Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:02.348736851Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:02.67707547Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:02.677787467Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:03.699484228Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:04.678135648Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:04.720464427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:07.179712659Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:07.213557269Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:07.240324535Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:07.240622644Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:07.260021962Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:07.261109412Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:08.009655576Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:08.579644361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:09.500233157Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:09.501051972Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:09.730839348Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:09.731373923Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:10.045024846Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:10.433066326Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:49:10.861093243Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:12.440695481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:12.441320271Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:12.519597071Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:14.846602044Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:16.714903618Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:16.715510388Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:17.044421663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:17.485147309Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:17.48558714Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:18.172560902Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:19.40930542Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:19.731565571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:19.734088222Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:49:20.859434629Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:21.230548593Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:21.231055756Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:21.942009131Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:21.942212601Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:22.782810475Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:23.128216867Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:23.12855635Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:25.698483536Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:25.745209917Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:25.745536031Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:26.276293513Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:26.276697757Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:27.327571013Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:27.328020572Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:28.170533715Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:28.17139431Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:28.738558885Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:28.739017473Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:28.953360909Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:49:30.86379064Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:31.649950903Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:31.650541042Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:31.770690289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:31.771020307Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:32.301632644Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:32.302220048Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:32.383631509Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:32.384141169Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:32.599591207Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:32.600229301Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:32.905529409Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:33.214617256Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:33.215121908Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:33.795320443Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:33.796106293Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:33.825180075Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:33.826844508Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:33.830045162Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:34.150164486Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:34.150534425Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:35.310557519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:35.576956225Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:35.577441593Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:36.661640699Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:36.662217477Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:36.926534277Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:37.166394261Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:37.167009901Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:39.206556521Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:39.208221006Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:40.702313936Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:40.702600674Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:49:40.860730037Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:41.282049866Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:41.282446026Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:41.488418405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:41.509581023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:41.510108782Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:41.558126252Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:41.558453491Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:41.715435265Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:42.558101852Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:42.558550696Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:43.862289663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:43.998022157Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:44.33848117Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:45.138753652Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:45.139106741Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:45.358138869Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:45.359099021Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:45.725362243Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:45.726103918Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:45.9391895Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:45.940055546Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:46.375594304Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:46.602736642Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:46.603289134Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:46.982417592Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:47.060608439Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:47.060997272Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:48.285477725Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:48.558247116Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:48.643359705Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:48.644067895Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:49.201432933Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:49.560212391Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:49.560656682Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:49.651413617Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:49:50.85946282Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:51.714893363Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:51.715597573Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:51.905215434Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:51.905777763Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:51.984038221Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:52.070460647Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:52.070890433Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:52.577240839Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:52.691162886Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:53.001693775Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:53.002130184Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:53.328189929Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:53.897087636Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:54.99717967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:54.997414849Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:55.305239614Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:55.305700449Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:55.412341961Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:55.412484694Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:55.754649227Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:55.755112591Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:56.402382602Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:58.265173191Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:58.88565398Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:59.302703735Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:59.30312331Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:59.712997397Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:59.713266885Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:59.870386033Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:49:59.870520058Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:50:00.859441178Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:00.930220127Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:01.045237145Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:01.045767321Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:02.61205992Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:02.662915851Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:03.899029905Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:03.899710086Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:03.900536561Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:03.961156813Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:04.680317925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:04.723834087Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:07.178656189Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:07.212367591Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:07.238863183Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:07.239161101Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:07.261346209Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:07.26214648Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:08.020823569Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:08.269511797Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:09.034612207Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:09.035028483Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:09.724919551Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:09.725491369Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:10.040263052Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:10.450592456Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:50:10.85941474Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:12.441983917Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:12.442549886Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:12.598154056Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:14.843911142Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:16.845535117Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:16.846166813Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:17.076900802Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:17.483786338Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:17.484178186Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:18.442602866Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:19.412997925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:20.002604175Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:20.003185242Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:50:20.859443409Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:21.226289651Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:21.22665592Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:21.900151835Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:21.900311501Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:22.766632005Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:23.116766428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:23.11733989Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:25.708483731Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:25.754261599Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:25.754839461Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:26.278959544Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:26.279457472Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:27.380646936Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:27.380933303Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:28.162906566Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:28.163517844Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:28.763135704Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:28.763578227Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:28.945630044Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:50:30.859419955Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:31.61482003Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:31.615377425Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:31.769335474Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:31.769531332Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:32.293589158Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:32.294102192Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:32.48383528Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:32.484223533Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:32.593369511Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:32.593964831Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:32.905513956Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:33.216184079Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:33.216458131Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:33.795328868Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:33.796148127Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:33.818941487Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:33.820177057Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:33.822647139Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:34.151266816Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:34.151556077Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:35.308412902Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:35.594784578Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:35.595208993Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:36.606185274Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:36.606535673Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:36.923785166Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:37.160971708Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:37.161444108Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:39.602932162Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:39.603431242Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:40.705797319Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:40.706071972Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:50:40.859490878Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:41.247539927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:41.24786926Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:41.488085944Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:41.523709156Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:41.524088096Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:41.55620013Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:41.556480905Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:41.719088721Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:42.545419645Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:42.545863564Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:43.870432101Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:44.005126264Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:44.337830889Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:45.1386242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:45.138947358Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:45.358214408Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:45.3590885Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:45.734794489Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:45.735511996Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:45.935383141Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:45.936270714Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:46.375133291Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:46.603445529Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:46.603813435Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:46.983731512Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:47.064343141Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:47.064825319Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:48.294618026Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:48.558477402Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:48.676168101Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:48.689365133Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:49.201021313Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:49.495135366Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:49.495428806Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:49.753251192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:50:50.859437988Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:51.709411134Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:51.710276014Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:51.944427055Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:51.945090973Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:51.984714577Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:52.078083205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:52.078488155Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:52.578324272Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:52.690823718Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:52.999016428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:52.999444831Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:53.338229638Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:53.895947379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:54.976852357Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:54.977077098Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:55.373789651Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:55.374327554Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:55.431874073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:55.432012963Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:55.789028226Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:55.789519163Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:56.419724384Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:58.260432066Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:58.853935111Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:59.092731649Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:59.09315793Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:59.715824203Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:59.71613727Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:59.888999752Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:50:59.88915286Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:51:00.859433639Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:00.931936649Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:01.149906436Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:01.150397059Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:01.789580131Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:02.370424076Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:02.370572284Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:02.66710069Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:02.667807885Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:03.698017817Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:04.674478207Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:04.739461988Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:07.178150771Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:07.213056722Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:07.239790555Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:07.240108723Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:07.263607007Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:07.264356497Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:08.017248826Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:08.617532343Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:08.834394811Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:08.834832277Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:09.696056573Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:09.696636076Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:10.012548127Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:10.567722666Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:51:10.859499237Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:12.472260021Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:12.4728174Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:12.518007303Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:14.855210464Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:16.72882526Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:16.729293511Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:17.079341138Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:17.421099959Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:17.421443769Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:18.133317031Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:19.408622877Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:19.697771865Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:19.699050994Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:51:20.859420788Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:21.214639041Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:21.215015887Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:21.917554064Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:21.917729994Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:22.762969056Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:23.116213664Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:23.116447259Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:25.697776855Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:25.744955565Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:25.745218727Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:26.272680466Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:26.272976375Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:27.380342487Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:27.380626857Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:28.170171322Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:28.170632507Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:28.761287359Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:28.767958269Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:28.936164705Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:51:30.859438799Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:31.64295861Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:31.643507607Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:31.770366434Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:31.770597489Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:32.307465488Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:32.307967928Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:32.481899935Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:32.482270682Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:32.593041941Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:32.593487805Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:32.905452106Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:33.215368985Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:33.215719703Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:33.79533338Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:33.796073287Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:33.816066511Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:33.817202318Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:33.819359801Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:34.164593773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:34.164871869Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:35.311216265Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:35.576859582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:35.577316921Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:36.646746486Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:36.647229669Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:36.921480161Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:37.170243121Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:37.170700687Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:39.362626039Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:39.363186273Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:40.685819572Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:40.686067324Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:51:40.859437271Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:41.27126274Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:41.271599529Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:41.415349925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:41.415706275Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:41.486851246Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:41.570016843Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:41.57045075Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:41.719465975Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:42.209773814Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:42.210200733Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:43.862307632Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:44.004205628Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:44.339816554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:44.972255648Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:44.972536558Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:45.358143896Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:45.358898894Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:45.733582906Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:45.734267351Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:45.927355275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:45.927977733Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:46.374838809Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:46.7131307Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:46.713484155Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:46.986246685Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:47.058218807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:47.058515554Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:48.26151418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:48.553535171Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:48.642606182Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:48.643238625Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:49.200849044Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:49.493725269Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:49.493954624Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:49.652394428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:51:50.859439836Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:51.701709542Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:51.702383392Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:51.917049018Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:51.917730938Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:51.985886488Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:52.065067676Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:52.065418196Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:52.584377342Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:52.690740277Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:52.998940468Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:52.999371156Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:53.33076042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:53.897051314Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:54.979963066Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:54.980163501Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:55.333422431Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:55.333931464Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:55.411986157Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:55.412137736Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:55.716098828Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:55.716543607Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:56.40252687Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:58.271256512Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:58.859440481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:59.287135832Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:59.287473119Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:59.713739875Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:51:59.714086241Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:00.05214865Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:00.05231782Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:52:00.859413638Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:00.93031885Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:01.035422143Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:01.035951254Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:01.722967271Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:02.381477534Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:02.381639338Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:02.684376998Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:02.685423084Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:03.702864449Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:04.680297571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:04.725716765Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:07.173538729Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:07.214602784Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:07.235172958Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:07.235396324Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:07.300291822Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:07.301195179Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:08.016227082Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:08.58980168Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:09.404811499Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:09.405086972Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:09.691676774Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:09.692091949Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:10.012273304Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:10.43424836Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:52:10.859779524Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:12.443943752Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:12.444523639Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:12.595344839Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:14.841865343Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:16.715092594Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:16.715555284Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:17.045051925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:17.453841821Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:17.454233189Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:18.2183157Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:19.4090648Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:19.707449972Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:19.707978507Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:52:20.859437722Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:21.208037836Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:21.208465173Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:21.939409487Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:21.93958085Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:22.774491199Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:23.116125426Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:23.116370872Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:25.697366213Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:25.745646614Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:25.745932504Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:26.29395209Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:26.294410362Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:27.399683239Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:27.40019476Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:28.16057821Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:28.161421011Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:28.567520954Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:28.568605278Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:28.974709664Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:52:30.859430368Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:31.672689112Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:31.673404416Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:31.769618357Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:31.769875493Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:32.28010379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:32.280553682Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:32.47862045Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:32.4790362Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:32.593534556Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:32.594155806Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:32.905612806Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:33.214168015Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:33.214500141Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:33.795357305Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:33.796073465Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:33.817377997Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:33.818509685Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:33.820693175Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:34.150475297Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:34.150774927Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:35.309188727Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:35.577043933Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:35.577534407Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:36.610158239Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:36.610515173Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:36.925915754Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:37.161819807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:37.162225526Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:39.311334558Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:39.311855276Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:40.685699707Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:40.685972426Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:52:40.859443587Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:41.079363248Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:41.079713838Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:41.417030964Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:41.417370547Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:41.487464092Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:41.554566361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:41.554925819Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:41.742646623Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:42.554587434Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:42.555159737Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:43.869160005Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:43.998057655Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:44.33919175Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:45.137403509Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:45.137768978Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:45.358158601Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:45.359012123Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:45.719326935Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:45.720007548Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:45.916049319Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:45.916605959Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:46.375789265Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:46.689901733Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:46.690265023Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:46.981086757Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:47.084520842Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:47.084857922Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:48.258953021Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:48.555167774Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:48.643192761Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:48.64362528Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:49.201014071Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:49.555800387Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:49.556039881Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:49.65042459Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:52:50.859424591Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:51.711160657Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:51.711881056Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:51.93380141Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:51.934331879Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:51.985942898Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:52.075698271Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:52.076097781Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:52.576571104Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:52.697821512Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:52.999101366Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:52.999547923Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:53.335563859Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:53.896067749Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:54.971248918Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:54.971468859Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:55.33263078Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:55.333082763Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:55.443569535Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:55.443748508Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:55.638027321Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:55.638442045Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:56.401790887Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:58.254307987Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:58.851727738Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:59.290173362Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:59.290512074Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:59.712724241Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:52:59.712957078Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:00.05409205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:00.054266863Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:53:00.859429819Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:00.930602317Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:01.032576651Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:01.033100177Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:01.670003149Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:02.397472802Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:02.39763257Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:02.695901173Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:02.696605476Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:03.698428158Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:04.674625938Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:04.719964334Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:07.171258256Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:07.202193403Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:07.223383932Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:07.223720331Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:07.246888609Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:07.2479941Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:08.016609314Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:08.04894429Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:09.391215678Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:09.39152862Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:09.681721239Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:09.682155558Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:10.028405932Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:10.456985222Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:53:10.880817295Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:12.44115996Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:12.441752586Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:12.515420375Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:14.841401668Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:16.724533197Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:16.724996092Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:16.949433967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:16.94979061Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:17.072463406Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:18.290423993Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:19.408072012Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:19.985331298Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:19.985813367Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:53:20.859434962Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:21.225044108Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:21.225517147Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:21.930013554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:21.930177018Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:22.757828023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:23.136448359Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:23.136726596Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:25.698072007Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:25.744932809Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:25.745234819Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:26.271988221Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:26.272304545Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:27.31573566Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:27.316059329Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:28.164602378Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:28.165429938Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:28.54004206Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:28.540468892Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:28.931622657Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:53:30.859437274Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:31.641821393Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:31.642383493Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:31.769210465Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:31.769448651Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:31.955607995Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:31.95609477Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:32.487589836Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:32.487971976Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:32.564587544Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:32.565066632Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:32.905570513Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:33.223745991Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:33.224093282Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:33.795273854Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:33.796027473Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:33.817469913Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:33.818613527Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:33.820848032Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:34.150620202Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:34.150872473Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:35.310219198Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:35.576870124Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:35.57737629Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:36.630441293Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:36.630956133Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:36.921236208Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:37.173445847Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:37.173908289Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:39.142626914Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:39.143066365Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:40.685636127Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:40.68592948Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:53:40.859418173Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:41.258633832Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:41.258974366Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:41.487370347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:41.53073873Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:41.531133111Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:41.569178715Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:41.569475543Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:41.719867133Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:42.222443477Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:42.22284931Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:43.859359041Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:43.997456403Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:44.33898795Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:44.564737019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:44.565049559Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:45.358081905Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:45.358899559Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:45.729433515Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:45.730087579Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:45.925726527Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:45.926337025Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:46.374494368Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:46.599594764Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:46.599951407Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:46.985281075Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:47.078977221Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:47.07931841Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:48.266616994Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:48.510967827Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:48.511484262Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:48.55362461Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:49.201825293Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:49.557044138Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:49.557268438Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:49.645504068Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:53:50.859403325Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:51.702892044Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:51.703844427Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:51.905166035Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:51.905680414Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:51.987698088Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:52.082057619Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:52.082454616Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:52.577053347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:52.692920133Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:52.998972395Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:52.999431548Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:53.33035568Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:53.898368995Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:54.972547131Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:54.972777091Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:55.33317683Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:55.333620539Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:55.412088191Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:55.412204656Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:55.640147234Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:55.640545524Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:56.406506271Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:58.266105577Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:58.851434925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:59.277343693Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:59.277691802Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:59.712653456Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:59.712944357Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:59.913486338Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:53:59.913622952Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:54:00.859441112Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:00.930825199Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:01.017929824Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:01.018353532Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:02.080788827Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:02.352127111Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:02.352297347Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:02.670770091Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:02.671461264Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:03.697356998Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:04.681086702Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:04.720067649Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:07.190707547Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:07.228474018Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:07.251349047Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:07.251528876Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:07.26560858Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:07.266487596Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:08.012011639Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:08.517866997Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:09.373122285Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:09.373411335Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:09.682137538Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:09.682541753Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:10.01739604Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:10.548420509Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:54:10.859752531Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:12.44407944Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:12.445133098Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:12.51714975Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:14.844696635Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:16.847746432Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:16.848127665Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:16.850102637Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:16.850631674Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:17.043946147Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:18.146614137Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:19.407025724Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:19.68856635Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:19.689075257Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:54:20.859436008Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:21.207008206Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:21.207342835Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:21.90835175Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:21.908520472Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:22.773936309Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:23.141128133Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:23.141368554Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:25.698734665Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:25.747564612Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:25.747870995Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:26.29917532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:26.299443812Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:27.383455618Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:27.383756942Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:28.170140173Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:28.170900589Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:28.578253344Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:28.579007043Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:28.961292015Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:54:30.859440101Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:31.617147188Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:31.617688583Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:31.769775064Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:31.770029053Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:31.955191732Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:31.955595384Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:32.36548798Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:32.365992989Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:32.59234594Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:32.592823975Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:32.905575894Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:33.213859684Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:33.214173268Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:33.795252858Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:33.795978302Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:33.815314241Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:33.816444033Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:33.818592646Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:34.151111579Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:34.15138768Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:35.30945621Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:35.585823585Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:35.586249074Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:36.633882428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:36.634253845Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:36.920916681Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:37.185535291Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:37.185995077Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:39.19508106Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:39.195736264Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:40.685947315Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:40.686221246Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:54:40.859482938Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:41.244124554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:41.24440891Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:41.487396136Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:41.518542361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:41.518946791Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:41.556879398Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:41.55718487Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:41.708978494Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:42.535448351Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:42.535929049Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:43.85833479Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:43.997872622Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:44.337481241Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:45.11943836Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:45.119768756Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:45.358124518Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:45.35890482Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:45.751533836Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:45.752241453Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:45.924060125Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:45.924706491Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:46.374363276Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:46.59540084Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:46.595770927Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:46.984920516Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:47.227597777Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:47.228058931Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:48.259526957Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:48.511528292Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:48.511999794Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:48.554212688Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:49.200314991Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:49.555390865Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:49.555623457Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:49.639687789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:54:50.859489777Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:51.701034844Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:51.701756775Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:51.915026668Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:51.915703096Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:51.986928166Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:52.070316012Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:52.070692169Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:52.593599396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:52.693082692Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:52.998813394Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:52.999205266Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:53.334026537Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:53.896543268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:55.004236155Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:55.004486168Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:55.338120686Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:55.33860889Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:55.471350714Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:55.471530886Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:55.763464521Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:55.763962009Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:56.402977658Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:58.259880625Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:58.851402454Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:59.276689385Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:59.276985976Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:59.714071641Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:54:59.71432456Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:00.048487713Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:00.048648308Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:55:00.859471467Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:00.859534101Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:00.86001378Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:00.932204569Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:02.553349809Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:02.553540971Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:02.82408194Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:02.888009127Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:02.889089597Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:03.713492Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:04.676839173Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:04.720220182Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:07.172242417Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:07.202951793Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:07.222576211Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:07.222863173Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:07.244321262Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:07.245342154Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:08.022881348Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:09.121069311Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:09.52045402Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:09.530764854Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:09.73150016Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:09.741984121Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:10.020650286Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:10.433344402Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:55:10.859508711Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:12.440137766Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:12.440727293Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:12.595480822Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:14.840865887Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:16.840522105Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:16.840990917Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:17.077965376Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:17.455398432Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:17.455710679Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:18.4926812Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:19.408272819Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:20.004540373Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:20.005091821Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:55:20.859436229Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:21.219928378Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:21.220290209Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:21.937064611Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:21.937246818Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:22.758190757Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:23.140364674Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:23.14099793Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:25.71853951Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:25.744594359Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:25.744876256Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:26.340587803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:26.341016307Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:27.379526438Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:27.379815889Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:28.159533069Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:28.160103471Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:28.770442693Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:28.771078139Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:28.947193551Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:55:30.859421732Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:31.634509464Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:31.635232963Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:31.76973919Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:31.769982008Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:32.280697559Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:32.281113651Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:32.348551017Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:32.349059842Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:32.593284573Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:32.593761015Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:32.905710816Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:33.21394514Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:33.214265379Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:33.795295627Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:33.796083611Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:33.816164239Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:33.817391416Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:33.81973041Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:34.15068438Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:34.15092667Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:35.312923465Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:35.576961984Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:35.577432338Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:36.60746405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:36.607936232Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:36.920454707Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:37.161000228Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:37.161441998Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:39.22183672Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:39.222330576Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:40.685911983Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:40.686166512Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:55:40.859434672Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:41.142976267Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:41.143291997Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:41.429374973Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:41.429780279Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:41.486947536Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:41.561014451Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:41.561416794Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:41.726502825Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:42.549963304Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:42.550426583Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:43.856956224Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:43.997748289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:44.272903108Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:45.112810069Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:45.113078546Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:45.358216659Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:45.35947258Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:45.714001602Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:45.714687389Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:45.938300539Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:45.938952035Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:46.374703316Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:46.772495987Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:46.772943036Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:46.982609871Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:47.22452566Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:47.224860246Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:48.26168944Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:48.512089233Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:48.512776342Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:48.557003229Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:49.209187476Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:49.556574938Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:49.562961037Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:49.639131291Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:55:50.85943117Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:51.716886066Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:51.717780713Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:51.913619391Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:51.914111836Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:51.98808824Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:52.067637282Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:52.068027561Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:52.578868057Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:52.692644498Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:52.999933251Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:53.000404814Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:53.340258576Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:53.90028893Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:54.972910973Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:54.973093854Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:55.296340623Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:55.297085004Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:55.429917001Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:55.430101481Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:55.735567263Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:55.735971327Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:56.409553774Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:58.256490624Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:58.855800558Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:59.310994288Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:59.311400903Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:59.713182455Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:59.714624923Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:59.889620297Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:55:59.889767535Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:56:00.859432973Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:00.931237871Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:00.984364227Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:00.984900541Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:02.151270658Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:02.347710842Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:02.347881768Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:02.7778935Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:02.778568762Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:03.69921597Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:04.679074817Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:04.721904398Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:07.174461149Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:07.205557268Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:07.224658483Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:07.224902667Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:07.249894827Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:07.250679068Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:08.015259478Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:09.259418075Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:09.486616306Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:09.486952787Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:09.818158191Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:09.82118925Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:10.019048214Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:10.431486204Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:56:10.859436169Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:12.45124046Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:12.451840176Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:12.515284306Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:14.843778335Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:16.724102304Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:16.72450073Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:17.076406909Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:17.470159872Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:17.470555818Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:18.311658262Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:19.405419257Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:19.696676983Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:19.69719396Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:56:20.859467587Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:21.352191653Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:21.352644549Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:21.925368061Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:21.925538948Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:22.772994237Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:23.134502221Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:23.134784417Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:25.699410293Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:25.744183683Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:25.744401824Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:26.272189391Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:26.27249142Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:28.1672103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:28.167837078Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:28.757085226Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:28.757619636Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:28.945514798Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:56:30.859866139Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:31.696356208Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:31.697834503Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:31.7733592Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:31.77387826Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:32.370242048Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:32.37076816Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:32.372548627Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:32.372957096Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:32.598395944Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:32.598882559Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:32.905554033Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:33.213982705Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:33.214280182Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:33.795302338Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:33.796014223Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:33.817282647Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:33.818391779Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:33.820696391Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:34.15211295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:34.152388939Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:35.309607248Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:35.589695147Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:35.590181224Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:36.614247432Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:36.61465461Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:36.94839478Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:37.176398974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:37.176877996Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:37.302840598Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:37.303096065Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:39.252512828Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:39.253026566Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:40.686015467Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:40.686295435Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:56:40.859442173Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:41.254515472Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:41.254862456Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:41.486539705Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:41.507759802Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:41.508080498Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:41.554620756Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:41.554999793Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:41.727116974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:42.549065481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:42.549503635Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:43.867441064Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:43.998225265Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:44.338794536Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:44.76680103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:44.76711371Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:45.35820719Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:45.359510838Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:45.728407069Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:45.729128749Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:45.923288733Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:45.923950294Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:46.375892459Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:46.605118627Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:46.605474466Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:46.982907534Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:47.21049829Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:47.21082483Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:48.257889298Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:48.555541775Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:48.655221306Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:48.656597518Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:49.202812086Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:49.512646615Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:49.512894189Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:49.65140511Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:56:50.859445226Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:51.745947477Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:51.746732079Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:51.905175891Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:51.905703802Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:51.983913985Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:52.071860534Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:52.072291377Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:52.57742841Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:52.690827813Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:52.999373458Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:52.999860522Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:53.31703302Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:53.897768036Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:54.971467221Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:54.971727391Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:55.30919027Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:55.309622877Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:55.412091792Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:55.412247898Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:55.743175191Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:55.743628614Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:56.406729314Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:58.260344827Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:58.852002792Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:59.106768564Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:59.107249546Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:59.713717773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:59.714004295Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:59.868947162Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:56:59.869104234Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:57:00.859451851Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:00.930442666Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:01.195856261Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:01.196519697Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:01.865408461Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:02.367675209Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:02.367852493Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:02.685496956Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:02.686478807Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:03.698995699Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:04.678093644Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:04.720487566Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:07.17857686Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:07.207580375Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:07.226137602Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:07.226379332Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:07.286115343Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:07.286934537Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:08.027026924Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:08.191936774Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:09.422334129Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:09.422821859Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:09.700880846Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:09.701360556Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:10.045784324Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:10.443857642Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:57:10.859427486Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:12.443883478Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:12.444500764Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:12.516971486Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:14.84136354Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:16.71459832Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:16.715071939Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:17.04188211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:17.449381764Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:17.449696704Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:18.150383539Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:19.522005625Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:19.688058223Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:19.688557842Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:57:20.859429723Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:21.264238369Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:21.264567763Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:21.93724956Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:21.937427597Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:22.757959694Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:23.145290925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:23.145556157Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:25.698825308Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:25.747125579Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:25.747336155Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:26.271996549Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:26.272300888Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:27.381123307Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:27.381414739Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:28.170183599Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:28.171142695Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:28.730116975Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:28.730643307Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:28.947897585Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:57:30.859417483Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:31.769404643Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:31.7698129Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:32.31397702Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:32.314480423Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:32.486256156Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:32.486623953Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:32.491863256Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:32.492323551Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:32.604045872Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:32.604627725Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:32.905712897Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:33.213958355Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:33.214229374Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:33.795258521Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:33.795986732Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:33.815739961Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:33.816849471Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:33.819046763Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:34.150381008Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:34.150633282Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:35.309440155Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:35.576510138Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:35.577001182Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:36.659867847Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:36.660322674Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:36.953495198Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:37.160360177Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:37.160802763Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:39.236177354Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:39.236652156Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:40.704646574Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:40.704942841Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:57:40.85941391Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:41.266608008Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:41.266965829Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:41.490642056Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:41.511257514Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:41.511587289Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:41.559615404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:41.559963084Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:41.726240073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:42.565716149Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:42.56627588Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:43.856422671Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:43.997913996Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:44.339885998Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:45.127830092Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:45.128084902Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:45.358084275Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:45.358879857Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:45.724694248Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:45.725657846Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:45.924513091Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:45.925294542Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:46.375038798Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:46.59220138Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:46.593032419Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:46.980378677Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:47.073878499Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:47.074231544Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:48.256306811Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:48.558047684Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:48.820604853Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:48.82118051Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:49.202454246Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:49.493027355Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:49.493255614Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:49.659353182Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:57:50.859435891Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:51.7033053Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:51.704023586Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:51.915745067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:51.91635113Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:51.990260417Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:52.069483967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:52.069925779Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:52.576398404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:52.692858959Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:53.002257813Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:53.002709211Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:53.300332345Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:53.897051145Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:54.980597807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:54.980817175Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:55.296091769Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:55.296582402Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:55.471244534Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:55.4714211Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:55.741179984Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:55.741580424Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:56.403436757Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:58.28446427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:58.856235888Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:59.095150713Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:59.095469989Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:59.715215584Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:59.715503362Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:59.897553584Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:57:59.897730521Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:58:00.859439646Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:00.930390275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:01.00328204Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:01.003828938Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:01.787469153Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:02.375525672Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:02.375700043Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:02.769171741Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:02.77030839Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:03.698347213Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:04.69125197Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:04.719987131Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:07.180296959Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:07.209905196Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:07.229596995Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:07.229825123Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:07.248125784Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:07.248882571Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:08.013363494Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:08.350734054Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:09.219421036Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:09.219748967Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:09.691289149Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:09.691807191Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:10.010134968Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:10.431334931Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:58:10.859434566Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:12.442775951Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:12.443326526Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:12.525244939Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:14.840575072Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:16.840290915Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:16.840771575Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:17.079721892Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:17.464683178Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:17.465015446Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:18.207947399Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:19.410910625Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:19.683733151Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:19.684232376Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:58:20.85943031Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:21.206032932Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:21.206359147Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:21.908081581Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:21.908230538Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:22.774223159Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:23.137238156Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:23.13750418Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:25.697333563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:25.755888642Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:25.756124801Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:26.292926757Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:26.293223389Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:27.377984867Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:27.378250287Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:28.171935809Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:28.172616566Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:28.758933692Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:28.759478184Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:28.943196452Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:58:30.859454449Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:31.774410762Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:31.774639726Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:32.322991473Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:32.323992428Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:32.485859211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:32.48697005Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:32.508684571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:32.510014478Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:32.608806463Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:32.609607423Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:32.910376418Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:33.217822024Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:33.218386391Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:33.808703678Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:33.810200822Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:33.849278456Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:33.851607197Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:33.855937036Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:34.150609429Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:34.150895456Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:35.31180253Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:35.576994359Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:35.57744232Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:36.632520638Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:36.632951288Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:36.957947556Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:37.160801726Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:37.161252115Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:39.393154359Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:39.39363572Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:40.703532086Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:40.703875858Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:58:40.859502628Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:41.260587134Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:41.260928817Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:41.488014706Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:41.506718667Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:41.507043283Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:41.554361054Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:41.554644042Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:41.718267515Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:42.571406462Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:42.57184875Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:43.859188807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:43.997924105Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:44.338472198Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:44.559303233Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:44.559611343Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:45.358156434Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:45.358944414Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:45.730108261Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:45.730814951Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:45.935445418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:45.936131889Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:46.380650972Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:46.602335994Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:46.602744917Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:46.985014588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:47.07462715Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:47.074972589Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:48.258304513Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:48.55571679Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:48.64551358Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:48.646031462Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:49.201395195Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:49.504712264Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:49.504954602Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:49.651295608Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:58:50.859427275Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:51.708729208Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:51.709393085Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:51.904789691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:51.905302189Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:51.98536895Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:52.090996454Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:52.091436778Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:52.576807802Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:52.693556552Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:52.998850971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:52.999298527Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:53.314058883Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:53.898221346Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:54.979252139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:54.979464683Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:55.344226084Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:55.344710846Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:55.47040947Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:55.470577526Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:55.707785778Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:55.70823919Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:56.402137464Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:58.281514924Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:58.852783441Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:59.286348924Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:59.286729823Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:59.708257222Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:59.708514408Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:59.904034413Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:58:59.904194528Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:59:00.859440886Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:00.931745252Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:01.090088665Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:01.090585165Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:01.676509359Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:02.368644206Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:02.368800025Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:02.742157415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:02.742812352Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:03.697818606Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:04.720657906Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:04.77260724Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:07.182890102Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:07.22124084Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:07.247190865Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:07.247472472Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:07.260319615Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:07.261011315Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:08.011890138Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:08.286071307Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:08.978203904Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:08.978541632Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:09.725908203Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:09.726510707Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:10.012278484Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:10.431019424Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:59:10.859509613Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:12.441378502Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:12.442118555Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:12.596912652Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:14.849074441Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:16.84148307Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:16.841960219Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:17.04719972Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:17.426430468Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:17.426780576Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:18.190679493Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:19.408524123Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:19.989964691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:19.990485901Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:59:20.859524042Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:21.209584158Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:21.209958337Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:21.909469281Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:21.909646489Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:22.758498043Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:23.135210886Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:23.135487141Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:25.697894692Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:25.743811225Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:25.744034342Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:26.274056564Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:26.274390786Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:27.332813928Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:27.333068502Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:28.160330631Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:28.16120681Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:28.693966386Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:28.694384732Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:28.933292224Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:59:30.859409907Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:31.631715447Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:31.632330187Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:31.769469748Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:31.76974299Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:31.957503693Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:31.957915016Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:32.455681211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:32.456048484Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:32.592520313Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:32.592962995Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:32.905536726Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:33.213958179Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:33.214282729Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:33.795262136Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:33.796027629Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:33.816965639Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:33.818210568Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:33.8205762Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:34.150568912Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:34.15082966Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:35.311330576Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:35.581188967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:35.582023263Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:36.673927361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:36.6744162Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:36.925183121Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:37.167278239Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:37.167738744Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:39.240658652Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:39.241085997Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:40.713219033Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:40.713519351Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:59:40.85943665Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:41.245264658Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:41.245544661Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:41.490881189Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:41.530835662Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:41.531222811Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:41.561408362Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:41.561896728Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:41.723625229Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:42.555934803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:42.556400135Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:43.856143366Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:43.998145807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:44.337925992Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:45.137122118Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:45.137479996Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:45.358093043Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:45.358859737Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:45.730460266Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:45.731180424Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:45.931291392Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:45.931945211Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:46.376250607Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:46.606011823Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:46.606331745Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:46.986118466Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:47.086192319Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:47.086508953Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:48.257184064Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:48.554259822Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:48.655877843Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:48.656621633Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:49.201785381Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:49.555133227Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:49.555380804Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:49.636405953Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T04:59:50.859451544Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:51.701242402Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:51.701950169Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:51.903704161Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:51.904179025Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:51.989104221Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:52.089861174Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:52.090334666Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:52.576213924Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:52.691015558Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:52.998957392Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:52.999407113Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:53.300550333Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:53.896406252Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:54.975622792Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:54.975907533Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:55.338814705Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:55.339410975Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:55.471930934Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:55.472122346Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:55.691266053Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:55.691784932Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:56.404982175Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:58.281908263Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:58.853529923Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:59.289554669Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:59.289898966Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:59.714395532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:59.714780194Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:59.870594745Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T04:59:59.870769977Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:00:00.85943527Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:00.930690356Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:01.067984587Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:01.068470572Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:02.529487383Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:02.625924677Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:03.175316276Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:03.176462454Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:04.095951062Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:04.811202793Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:04.946071393Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:08.39575754Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:08.613770405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:08.614985199Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:08.759214328Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:08.817488877Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:08.913558429Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:08.953838612Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:08.954177642Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:09.765018122Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:09.765596671Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:10.04511228Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:10.33635798Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:10.341009271Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:10.483556897Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:00:10.884472432Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:12.486411475Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:12.487374432Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:12.595327398Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:13.882111075Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:14.889779654Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:16.753489499Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:16.754336448Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:16.961350028Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:16.962161245Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:17.084933011Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:18.513395601Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:19.519948686Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:19.757205246Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:19.757891629Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:00:20.859478517Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:21.252198775Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:21.25251097Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:21.919865608Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:21.920044926Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:22.772454581Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:23.409111435Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:23.409361476Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:25.697609655Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:25.745924965Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:25.74634852Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:26.272421332Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:26.272727433Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:27.380188241Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:27.380514672Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:28.173129668Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:28.174856192Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:28.674585645Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:28.676001403Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:28.960330795Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:00:30.859453859Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:31.641611278Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:31.642235435Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:31.769506777Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:31.769793491Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:31.973460955Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:31.973880489Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:32.54607354Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:32.546402897Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:32.593208534Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:32.59364872Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:32.905499218Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:33.224491513Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:33.224956398Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:33.795408978Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:33.796133558Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:33.815696959Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:33.81713139Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:33.820206357Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:34.155864986Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:34.156312688Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:35.321424846Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:35.589056427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:35.590092898Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:36.635952824Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:36.636415518Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:36.924757877Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:37.160804039Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:37.161183545Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:39.674962566Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:39.675402078Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:40.816205403Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:40.819248054Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:00:40.859387729Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:41.251139927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:41.251545475Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:41.487110427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:41.528497836Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:41.529167932Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:41.558693347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:41.559025319Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:41.688219473Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:42.549180407Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:42.549568718Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:43.95152165Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:44.003069972Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:44.338403607Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:44.936631036Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:44.936896979Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:45.358116872Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:45.358840513Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:45.730101673Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:45.730781054Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:45.913697748Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:45.914484904Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:46.375788047Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:46.708359377Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:46.708722449Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:46.984842424Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:47.079884817Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:47.080201187Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:48.273180827Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:48.558549526Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:48.645007938Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:48.645714805Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:49.200686649Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:49.557010847Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:49.557267093Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:49.643646978Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:00:50.85941979Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:51.734432936Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:51.735180161Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:51.906023162Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:51.906558365Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:51.984967673Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:52.076718718Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:52.077225944Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:52.580020479Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:52.691808095Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:52.999562294Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:53.000011838Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:53.314480691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:53.899389137Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:55.002995916Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:55.003504126Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:55.332911559Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:55.333806479Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:55.479305181Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:55.479574612Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:55.828431228Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:55.828888291Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:56.403232462Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:58.284067136Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:58.873104499Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:59.129421669Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:59.130080755Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:59.718554362Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:59.718902464Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:59.872338492Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:00:59.872490301Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:01:00.859455701Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:00.931912254Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:01.080766582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:01.081185226Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:01.861401341Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:02.346089027Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:02.34627453Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:02.689217043Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:02.689905573Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:03.699250011Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:04.683297908Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:04.744061637Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:07.190599245Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:07.235467274Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:07.263216888Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:07.26435527Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:07.295741645Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:07.296501404Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:08.015168755Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:09.477922672Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:09.478318729Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:09.698945763Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:09.699521298Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:10.026941724Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:10.441378133Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:01:10.859469172Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:11.143428328Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:12.44844976Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:12.449051683Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:12.596067562Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:14.848145482Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:16.716272555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:16.716738696Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:17.074014391Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:17.435123287Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:17.435438968Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:18.174894938Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:19.409016654Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:19.700479538Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:19.701013952Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:01:20.859462065Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:21.209733762Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:21.210085497Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:21.921253231Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:21.921439958Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:22.757719582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:23.141616857Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:23.141929801Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:25.697852551Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:25.746659128Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:25.746874667Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:26.29269522Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:26.293011542Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:27.384918291Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:27.385166523Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:28.164121793Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:28.164821472Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:28.723497358Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:28.724122102Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:28.933654538Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:01:30.859437938Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:31.624379007Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:31.624878031Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:31.770976329Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:31.771217842Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:32.275382602Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:32.275779456Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:32.345593265Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:32.345959049Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:32.59314281Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:32.593610819Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:32.905509133Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:33.213845067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:33.214179836Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:33.795271963Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:33.796019176Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:33.822116015Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:33.826356527Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:33.830813514Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:34.150011008Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:34.150261356Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:35.311112214Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:35.761961843Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:35.762420995Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:36.747846875Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:36.748320426Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:36.920845343Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:37.160883797Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:37.161320371Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:39.52950271Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:39.530399174Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:40.77020118Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:40.770587091Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:01:40.859447594Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:41.263883472Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:41.264167174Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:41.487273333Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:41.533021713Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:41.533408997Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:41.568047253Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:41.568350371Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:41.743921238Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:42.22157139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:42.22201871Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:43.859838473Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:44.01242939Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:44.339706916Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:45.162713163Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:45.163034259Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:45.358074026Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:45.35886463Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:45.715240496Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:45.715926252Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:45.926412663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:45.927027045Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:46.379926836Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:46.639328671Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:46.641507977Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:46.985635461Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:47.059826986Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:47.060171397Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:48.253790481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:48.553382755Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:48.648876467Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:48.649440964Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:49.201655818Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:49.556345106Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:49.556594702Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:49.646002047Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:01:50.859431305Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:51.702425555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:51.703094175Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:51.944104839Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:51.944761968Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:51.989701592Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:52.084888921Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:52.085401367Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:52.576796208Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:52.695193558Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:53.000437394Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:53.001051572Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:53.327229608Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:53.898314936Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:54.971835517Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:54.97204211Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:55.332733098Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:55.33318446Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:55.471976848Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:55.472155264Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:55.777886874Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:55.778304563Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:56.404960674Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:58.284546782Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:58.857260668Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:59.291226169Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:59.291613972Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:59.723930294Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:59.724235629Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:59.891475887Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:01:59.891718012Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:02:00.859448744Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:00.93311731Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:01.009290339Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:01.009767125Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:02.303474618Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:02.346318342Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:02.346469975Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:02.675768166Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:02.676377875Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:03.699466927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:04.682652813Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:04.7201848Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:07.176552785Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:07.206960816Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:07.225602495Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:07.225817061Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:07.272221344Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:07.272993976Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:08.010798418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:08.521328126Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:09.695727424Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:09.696192351Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:09.696626763Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:09.69698836Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:10.039437499Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:10.433161084Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:02:10.859488288Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:12.439948649Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:12.440497901Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:12.520923196Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:14.84490541Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:16.85175747Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:16.852223311Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:17.171260271Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:17.469809041Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:17.470223966Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:18.186697178Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:19.410748128Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:19.997887028Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:19.998378089Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:02:20.859444396Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:21.209854759Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:21.2101894Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:21.902082643Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:21.902265Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:22.776895723Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:23.135444373Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:23.135738385Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:25.697512151Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:25.743474261Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:25.743750858Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:26.286752277Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:26.287041662Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:28.16832134Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:28.169140812Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:28.555481257Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:28.555905689Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:28.938827029Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:28.964389323Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:28.964925345Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:02:30.859439857Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:31.620254423Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:31.620831412Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:31.770330086Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:31.770643392Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:32.297455299Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:32.29788288Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:32.355236085Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:32.355614487Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:32.592810963Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:32.593262324Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:32.905541309Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:33.213557447Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:33.213890659Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:33.795204371Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:33.795976853Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:33.815390637Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:33.816530691Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:33.818695456Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:34.160420852Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:34.160683341Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:35.309691229Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:35.576960292Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:35.577421581Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:36.677677685Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:36.678707256Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:36.922715196Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:37.164101274Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:37.165092109Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:39.332078151Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:39.332835272Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:40.687480485Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:40.687771319Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:02:40.859435406Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:41.081634672Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:41.081966318Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:41.487152879Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:41.534259404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:41.534703585Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:41.575209453Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:41.575517744Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:41.708985304Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:42.573774683Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:42.574385933Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:43.863523763Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:43.99835462Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:44.34075183Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:45.13749339Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:45.138100846Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:45.358122648Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:45.358885959Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:45.725119199Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:45.725780393Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:45.926461317Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:45.927127804Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:46.378930611Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:46.623982286Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:46.624327422Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:46.983362111Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:47.066714553Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:47.067110807Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:48.256304178Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:48.519325395Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:48.520168046Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:48.554985639Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:49.202225241Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:49.555302226Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:49.555552473Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:49.65161821Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:02:50.859433986Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:51.71054817Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:51.711224594Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:51.905867405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:51.906401557Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:51.991815383Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:52.089875075Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:52.090386233Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:52.577788614Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:52.692471274Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:52.998702699Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:52.999144579Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:53.30702067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:53.896369918Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:54.972712489Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:54.972940688Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:55.296036387Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:55.296507414Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:55.472074344Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:55.472238158Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:55.800733299Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:55.801162746Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:56.417076721Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:58.278006718Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:58.852808901Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:59.29150016Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:59.29184493Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:59.713817612Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:02:59.714088867Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:00.054291309Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:00.054482021Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:03:00.859445072Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:00.930746046Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:01.099918234Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:01.1004521Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:01.805414329Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:02.346153138Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:02.346317267Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:02.685092145Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:02.685829213Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:03.697768793Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:04.69672475Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:04.724043162Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:07.177256504Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:07.21028556Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:07.230710963Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:07.230960423Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:07.253130436Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:07.253876315Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:08.034876276Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:08.172581641Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:09.437558006Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:09.43837883Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:09.74500761Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:09.745565798Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:10.028101291Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:10.431302522Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:03:10.859421675Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:12.442171546Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:12.442816767Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:12.59561654Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:14.846221134Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:16.604070402Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:16.604361155Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:16.846098064Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:16.846576836Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:17.078279195Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:18.222467491Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:19.407235775Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:19.682602656Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:19.683150304Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:03:20.859436397Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:21.216496977Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:21.216824198Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:21.903770598Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:21.903907073Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:22.75846021Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:23.141209771Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:23.141431788Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:25.697458218Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:25.744123247Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:25.744346428Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:26.272027164Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:26.272300136Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:27.311227468Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:27.311490347Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:28.168266242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:28.169198726Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:28.710635529Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:28.711215326Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:28.933195562Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:03:30.859416974Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:31.620298192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:31.620961147Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:31.769103757Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:31.769329906Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:32.272020925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:32.272409478Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:32.348896035Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:32.34924743Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:32.593010196Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:32.593477962Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:32.905542456Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:33.213721789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:33.214055723Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:33.795353459Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:33.796279445Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:33.817052304Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:33.818183497Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:33.82040237Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:34.150236661Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:34.150475688Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:35.309510644Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:35.577222709Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:35.577907316Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:36.621080019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:36.621647964Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:36.924738444Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:37.160885009Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:37.161290295Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:39.122840856Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:39.123219349Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:40.700516237Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:40.700773584Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:03:40.859430879Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:41.259603199Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:41.259913221Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:41.487262797Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:41.535519361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:41.535981615Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:41.574202971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:41.574525038Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:41.714648725Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:42.591708831Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:42.59226601Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:43.857888243Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:43.997908877Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:44.337982898Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:44.566047209Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:44.566369838Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:45.358099949Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:45.358892449Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:45.714335533Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:45.715036508Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:45.917024073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:45.917576021Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:46.376277025Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:46.619156616Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:46.619500588Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:46.985620387Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:47.073040178Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:47.073391355Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:48.254297662Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:48.511151283Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:48.511582099Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:48.554091475Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:49.200996027Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:49.493384952Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:49.493630238Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:49.638974801Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:03:50.859451291Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:51.71150832Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:51.712194701Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:51.923912716Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:51.924572087Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:51.989166471Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:52.073612771Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:52.074015498Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:52.576896354Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:52.690703523Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:52.998842964Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:52.99928219Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:53.314434204Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:53.898208227Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:54.973806128Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:54.974026517Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:55.30519057Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:55.305637698Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:55.482168814Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:55.48234804Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:55.773117519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:55.773547867Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:56.403857834Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:58.296078678Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:58.852633901Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:59.28882477Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:59.289142675Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:59.711876545Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:59.712165608Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:59.88710702Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:03:59.887276854Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:04:00.85944081Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:00.930877611Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:01.105109028Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:01.105641593Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:02.426777906Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:02.426943711Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:02.462777243Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:02.764875644Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:02.76551996Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:03.701257519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:04.685610189Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:04.720954884Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:07.18431987Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:07.221237281Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:07.247735877Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:07.248004619Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:07.265678371Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:07.266405869Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:08.012803316Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:09.093285878Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:09.500022442Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:09.500419117Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:09.722282566Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:09.722925415Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:10.01854476Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:10.432187105Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:04:10.859420742Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:12.441552046Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:12.442128287Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:12.596794821Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:14.854398396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:16.716521622Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:16.717024613Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:17.074459699Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:17.449453492Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:17.449804196Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:18.360724522Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:19.408259927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:19.687205028Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:19.687754751Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:04:20.859425006Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:21.207767943Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:21.208093531Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:21.924301731Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:21.924464051Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:22.75914135Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:23.133957286Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:23.134206567Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:25.702748348Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:25.746380445Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:25.746679515Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:26.29748264Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:26.297790551Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:27.391950146Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:27.392735905Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:28.159659343Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:28.160267182Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:28.776529437Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:28.777091958Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:28.968300804Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:04:30.859416357Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:31.651713724Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:31.65229321Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:31.769753063Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:31.769998292Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:32.287321842Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:32.287766165Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:32.449963616Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:32.450332879Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:32.59345023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:32.593920064Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:32.90572253Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:33.231324682Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:33.231649174Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:33.795396184Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:33.796103378Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:33.816275449Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:33.817382235Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:33.81956517Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:34.150620247Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:34.150903147Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:35.312770719Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:35.577890261Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:35.578317448Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:36.649737597Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:36.650286726Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:36.920295124Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:37.172961415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:37.173384217Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:39.370116199Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:39.371343686Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:40.770847464Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:40.771179994Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:04:40.859778744Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:41.257816329Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:41.258167498Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:41.489893465Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:41.507066036Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:41.507383141Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:41.55954616Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:41.560024949Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:41.736947246Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:42.557534102Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:42.557986383Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:43.856873036Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:43.999110832Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:44.272578128Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:45.117866782Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:45.118188743Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:45.358123163Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:45.358902318Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:45.724818238Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:45.725522213Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:45.929835323Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:45.930556909Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:46.376402447Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:46.60069854Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:46.600995341Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:46.981367772Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:47.078008938Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:47.078284086Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:48.264736683Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:48.511647214Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:48.512311636Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:48.554190612Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:49.204636481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:49.492748999Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:49.492985087Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:49.652968246Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:04:50.859430226Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:51.710316972Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:51.711064893Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:51.912086473Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:51.912774109Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:51.986632481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:52.102360575Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:52.102872554Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:52.63211247Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:52.692451971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:53.002147723Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:53.002581392Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:53.30144283Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:53.89739611Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:54.981771184Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:54.981971409Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:55.296078682Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:55.296530543Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:55.413116271Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:55.41325984Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:55.668912308Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:55.669481192Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:56.402899283Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:58.254758035Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:58.856582665Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:59.275229541Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:59.275488012Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:59.709324041Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:59.709561933Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:59.889626388Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:04:59.889795804Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:05:00.859424365Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:00.93030669Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:01.002183046Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:01.00269134Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:02.348862899Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:02.472999426Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:02.473211827Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:02.778999962Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:02.77975136Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:03.701588207Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:04.68008685Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:04.72133529Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:07.212275588Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:07.259607593Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:07.271743386Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:07.272803093Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:07.282946673Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:07.283240325Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:08.019947446Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:08.793288596Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:09.569303512Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:09.56974164Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:09.682071804Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:09.682892438Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:10.012167362Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:10.430545323Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:05:10.85951033Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:12.440422752Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:12.441053695Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:12.515802801Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:14.842081737Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:16.714657284Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:16.715479485Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:17.076175427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:17.489636939Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:17.490048393Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:18.359874952Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:19.40753599Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:19.994388315Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:19.99493261Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:05:20.859466909Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:21.222449227Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:21.222818724Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:21.911455711Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:21.911702105Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:22.773214958Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:23.139772927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:23.140026684Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:25.697510825Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:25.744053874Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:25.744277738Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:26.275692819Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:26.275984362Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:27.394187425Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:27.39467387Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:28.16845373Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:28.169141803Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:28.523914977Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:28.524357263Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:28.931695996Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:05:30.859410469Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:31.769974902Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:31.770208682Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:31.965367209Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:31.965781038Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:32.465383866Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:32.465929284Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:32.471734142Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:32.47208018Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:32.573011176Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:32.57345503Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:32.905774544Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:33.213889222Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:33.214211392Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:33.795256073Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:33.795998749Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:33.816144598Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:33.817264935Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:33.819452847Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:34.150562882Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:34.150800721Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:35.31020783Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:35.581007279Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:35.581507143Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:36.634009975Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:36.634458372Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:36.920382394Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:37.161360173Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:37.161952997Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:39.149844121Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:39.150213706Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:40.738530356Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:40.740962671Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:05:40.859796562Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:41.262283278Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:41.26258218Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:41.439264604Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:41.439601242Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:41.487496965Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:41.565463232Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:41.566005662Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:41.715168273Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:42.554799736Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:42.555262473Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:43.858712741Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:43.997709351Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:44.337979884Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:45.140174093Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:45.140499745Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:45.358106497Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:45.358908471Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:45.729522297Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:45.730147707Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:45.926836158Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:45.927409053Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:46.374988581Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:46.765780182Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:46.766229931Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:46.983605935Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:47.06221377Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:47.062517041Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:48.264296629Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:48.55384407Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:48.643467917Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:48.644122028Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:49.205492368Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:49.557265173Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:49.557500656Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:49.665904527Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:05:50.859518154Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:51.721386358Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:51.722526202Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:51.895840125Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:51.896275581Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:52.066061946Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:52.066483832Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:52.080709851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:52.576982869Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:52.703599043Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:52.998833687Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:52.999221595Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:53.320276612Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:53.899390451Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:54.97382884Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:54.974060768Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:55.295972923Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:55.296398502Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:55.41296786Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:55.413106112Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:55.656047289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:55.656526221Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:56.401809504Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:58.263308784Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:58.861644429Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:59.277857359Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:59.278155967Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:59.708345045Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:59.708604392Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:59.886875474Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:05:59.887009529Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:06:00.859454366Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:00.931384459Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:00.970164942Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:00.970622136Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:02.050764017Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:02.368745315Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:02.368896186Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:02.6746436Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:02.675332301Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:03.700536325Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:04.681910075Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:04.721791909Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:07.185717857Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:07.218882528Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:07.245222834Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:07.24546841Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:07.262020092Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:07.262722949Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:08.04936002Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:09.081633411Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:09.411192965Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:09.411504739Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:09.711183736Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:09.71210322Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:10.083489332Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:10.432408718Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:06:10.859475854Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:12.440152555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:12.440754264Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:12.518153556Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:14.841391282Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:16.841821206Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:16.84227563Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:17.070801769Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:17.471116878Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:17.471483238Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:18.275221209Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:19.410604391Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:19.680803148Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:19.681308331Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:06:20.859488389Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:21.226577798Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:21.226936322Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:21.920657843Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:21.920841296Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:22.764321158Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:23.139430854Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:23.13969349Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:25.698209665Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:25.743761275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:25.744004875Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:26.273204159Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:26.273706173Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:27.31188852Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:27.31215363Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:28.16390994Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:28.16437734Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:28.612843263Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:28.613789488Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:28.941589531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:06:30.859424352Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:31.769735143Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:31.769994989Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:31.963896453Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:31.964274637Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:32.476297576Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:32.477030856Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:32.478906866Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:32.479216896Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:32.593700506Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:32.594301784Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:32.905714099Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:33.215831934Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:33.216164171Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:33.795291181Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:33.796052181Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:33.816352032Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:33.817464823Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:33.819632151Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:34.150220283Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:34.150454687Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:35.310463783Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:35.577377183Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:35.577840775Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:36.59906235Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:36.599443622Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:36.921511753Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:37.160749164Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:37.161205332Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:39.223224756Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:39.224083729Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:40.699001845Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:40.699515409Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:06:40.860732469Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:41.261581663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:41.262403655Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:41.491220825Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:41.55382943Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:41.554603849Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:41.607572822Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:41.608296367Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:41.717479925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:42.549602319Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:42.550057359Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:43.858194053Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:44.000818024Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:44.337909062Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:44.751658691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:44.751989935Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:45.358102757Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:45.358825152Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:45.731032968Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:45.731740125Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:45.942955471Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:45.943548534Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:46.378910151Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:46.605936367Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:46.60631767Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:46.986041758Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:47.218485233Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:47.218825185Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:48.257591853Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:48.553655813Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:48.650989942Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:48.651574785Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:49.202305808Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:49.556518355Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:49.556803212Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:49.636906225Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:06:50.859440921Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:51.704442942Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:51.705305278Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:51.902935808Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:51.903423574Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:51.984242439Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:52.072423711Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:52.072872515Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:52.57864601Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:52.691151481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:52.998972666Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:52.999389514Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:53.329917461Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:53.896120058Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:54.97165466Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:54.971896365Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:55.332645543Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:55.333114478Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:55.42961617Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:55.429801817Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:55.774675065Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:55.775106751Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:56.405587872Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:58.258459102Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:58.851625848Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:59.113658511Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:59.114101117Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:59.71579793Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:59.716068181Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:59.886948055Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:06:59.887103882Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:07:00.859426571Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:00.930399386Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:01.084257588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:01.084729948Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:01.987741209Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:02.385547437Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:02.385712164Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:02.674235762Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:02.674931493Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:03.698150849Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:04.720023182Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:04.769754829Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:07.181867033Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:07.214965856Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:07.241656516Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:07.241985511Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:07.264559003Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:07.265330547Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:08.03150324Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:08.163256071Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:08.847637429Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:08.84806458Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:09.732764853Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:09.733778608Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:10.018572941Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:10.430379012Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:07:10.859455205Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:12.483488788Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:12.484107692Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:12.515761471Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:14.844718395Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:16.840278497Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:16.840727281Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:17.079161938Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:17.426609902Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:17.427007919Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:18.130801719Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:19.408209426Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:19.693650664Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:19.69424233Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:07:20.859432839Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:21.244613026Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:21.244983013Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:21.901423547Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:21.901589668Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:22.758211073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:23.139155436Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:23.139462403Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:25.697883737Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:25.743615825Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:25.74387739Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:26.272358972Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:26.272644303Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:27.380640425Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:27.380960439Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:28.174525155Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:28.175195673Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:28.560224998Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:28.560783408Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:28.977634307Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:07:30.860496826Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:31.770038985Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:31.770274144Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:31.968016397Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:31.968512286Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:32.490420815Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:32.490883787Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:32.510166144Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:32.510614039Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:32.598434289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:32.598922042Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:32.905566535Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:33.213464224Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:33.21377127Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:33.795339439Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:33.796182088Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:33.816269091Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:33.817409012Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:33.819599313Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:34.156566628Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:34.156826936Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:35.310512501Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:35.585740817Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:35.586191672Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:36.617424347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:36.618095241Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:36.92144849Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:37.172322078Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:37.172914299Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:39.097379215Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:39.097906778Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:40.715946099Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:40.716273767Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:07:40.933194716Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:41.264600184Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:41.264961966Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:41.486368605Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:41.519652171Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:41.520030371Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:41.555977601Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:41.556300813Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:41.707157542Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:42.54762497Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:42.548053525Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:43.859923223Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:43.998481432Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:44.339259592Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:45.119725569Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:45.120050919Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:45.358078792Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:45.358827601Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:45.71601323Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:45.716732308Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:45.922011749Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:45.92260798Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:46.375796237Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:46.62449804Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:46.641070819Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:46.980239627Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:47.214169082Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:47.214488913Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:48.304440435Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:48.553879341Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:48.648254791Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:48.648941812Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:49.211569487Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:49.566320754Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:49.566552938Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:49.643481029Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:07:50.859428302Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:51.748990571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:51.749631902Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:51.905959769Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:51.906432312Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:51.989977426Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:52.080846782Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:52.081253375Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:52.588004137Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:52.690700329Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:53.002334207Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:53.002810881Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:53.331217117Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:53.898825255Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:54.979779686Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:54.980000366Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:55.336536157Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:55.33703504Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:55.429366067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:55.429525189Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:55.759181168Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:55.759627519Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:56.402217613Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:58.257433825Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:58.851387423Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:59.092355568Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:59.092758459Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:59.722961295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:59.723222371Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:59.885991507Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:07:59.886120598Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:08:00.859428632Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:00.933086105Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:01.080321731Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:01.080804211Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:01.78721574Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:02.478884624Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:02.479046074Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:02.672872354Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:02.67347498Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:03.703218756Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:04.675199894Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:04.723454747Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:07.187636585Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:07.21721641Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:07.237156538Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:07.237418213Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:07.264905748Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:07.265729536Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:08.010362433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:09.099883679Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:09.236875618Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:09.237282176Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:09.751494092Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:09.75302954Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:10.037242474Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:10.431394557Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:08:10.859424696Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:12.442356859Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:12.442980888Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:12.596087487Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:14.84081028Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:16.841329396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:16.84179751Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:17.044857054Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:17.442411258Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:17.442777098Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:18.138950127Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:19.407626947Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:19.993472315Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:19.993976389Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:08:20.859456126Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:21.21287531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:21.213218662Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:21.926097191Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:21.926291856Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:22.757521895Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:23.134768205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:23.135056254Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:25.697746346Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:25.747577911Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:25.747840435Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:26.27214701Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:26.272454015Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:27.379977219Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:27.380868271Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:28.170058246Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:28.170636063Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:28.766078409Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:28.766591342Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:28.961216803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:08:30.859462665Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:31.770592851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:31.770924536Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:32.284485212Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:32.284927421Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:32.373609933Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:32.373979791Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:32.460000622Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:32.460515019Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:32.59379756Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:32.594269393Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:32.905767278Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:33.218764103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:33.219038856Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:33.79525957Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:33.796013629Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:33.816292067Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:33.817425984Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:33.819616648Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:34.150779784Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:34.15109129Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:35.311562138Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:35.577330815Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:35.577823358Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:36.599541103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:36.599937059Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:36.942932893Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:37.198760929Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:37.19935337Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:39.120433252Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:39.120806714Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:40.718022261Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:40.718326191Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:08:40.859428653Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:41.244935112Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:41.245230182Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:41.48795247Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:41.525334029Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:41.525841543Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:41.560288687Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:41.560573193Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:41.728936535Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:42.245041218Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:42.24616053Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:43.863683801Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:43.997821131Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:44.276305516Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:44.549805699Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:44.550074645Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:45.358335187Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:45.359104357Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:45.734536759Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:45.735228829Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:45.913014621Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:45.913649194Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:46.374064323Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:46.589297292Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:46.589635041Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:46.983221218Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:47.205447874Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:47.205784824Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:48.258121381Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:48.553524917Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:48.643455545Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:48.64415096Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:49.200453054Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:49.55689971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:49.557159324Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:49.640372974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:08:50.859422201Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:51.704873923Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:51.705591539Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:51.911485427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:51.912214878Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:51.986341155Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:52.065012266Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:52.065423252Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:52.577852279Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:52.691599513Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:53.002184251Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:53.002639054Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:53.328368139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:53.897317641Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:54.980619683Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:54.980857653Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:55.332317971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:55.332812455Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:55.413012832Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:55.413181442Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:55.749172571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:55.749609673Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:56.417917786Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:58.273717909Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:58.852498933Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:59.293435219Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:59.293767382Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:59.712995978Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:59.713258727Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:59.891758416Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:08:59.891911269Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:09:00.859434171Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:00.930659367Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:00.99058472Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:00.991107881Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:01.51901455Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:02.357250899Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:02.357391492Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:02.67369805Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:02.674362922Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:03.791181204Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:04.67580916Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:04.719870105Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:07.177918702Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:07.212448253Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:07.238757735Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:07.239042426Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:07.264121836Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:07.264823161Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:08.009705257Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:08.114804182Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:09.536360543Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:09.536877389Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:09.788921777Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:09.789646531Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:10.025871624Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:10.432405747Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:09:10.859514947Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:12.445903033Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:12.446528132Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:12.516297021Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:14.843585209Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:16.724231485Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:16.72467487Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:17.187223433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:17.441626275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:17.441978731Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:18.134135875Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:19.407018379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:19.683027788Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:19.683520771Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:09:20.859463796Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:21.221050239Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:21.221414662Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:21.916183737Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:21.916354381Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:22.776858495Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:23.136605405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:23.137006723Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:25.698183467Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:25.753965497Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:25.754273005Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:26.310376813Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:26.310688877Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:27.379157599Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:27.379488426Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:28.170723958Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:28.171309861Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:28.694550289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:28.695270077Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:28.934081212Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:09:30.859435452Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:31.773025739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:31.773292566Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:32.277603406Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:32.278003838Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:32.345187707Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:32.345590672Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:32.480556974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:32.481079587Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:32.60439161Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:32.604957675Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:32.905569598Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:33.214098456Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:33.214435336Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:33.795239077Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:33.795978486Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:33.816451203Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:33.817991827Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:33.820297744Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:34.155343283Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:34.155647566Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:35.310334811Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:35.57659415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:35.57709971Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:36.641405093Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:36.641912557Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:36.920823286Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:37.168442235Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:37.169147064Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:39.178212514Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:39.17857213Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:40.685813996Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:40.686073559Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:09:40.859433997Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:41.297146722Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:41.297469404Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:41.490686863Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:41.512089624Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:41.51240314Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:41.571040415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:41.571376365Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:41.720140949Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:42.567287089Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:42.567740534Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:43.858914779Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:44.001152434Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:44.3437375Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:45.132496556Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:45.132833355Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:45.358083443Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:45.358819096Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:45.734522365Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:45.735259007Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:45.958602459Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:45.9592313Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:46.375351499Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:46.608636344Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:46.609088658Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:46.980810632Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:47.103794237Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:47.104115946Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:48.257777523Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:48.553376972Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:48.655497335Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:48.656256184Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:49.201270471Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:49.493413815Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:49.493619722Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:49.640374044Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:09:50.859438636Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:51.700061289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:51.700770618Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:51.914397315Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:51.915085005Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:51.983334078Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:52.076198378Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:52.076598161Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:52.577372629Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:52.691555663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:52.9987339Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:52.999208708Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:53.330543961Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:53.896364483Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:55.01475935Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:55.014994307Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:55.332564237Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:55.333341609Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:55.412143029Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:55.412298765Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:55.813447713Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:55.814133493Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:56.407142726Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:58.263973299Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:58.856248799Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:59.277147149Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:59.27740295Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:59.712793835Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:59.713063084Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:59.88735297Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:09:59.887516905Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:10:00.859438197Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:00.930532301Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:00.976832653Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:00.977323383Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:02.507108508Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:02.574458702Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:03.064347411Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:03.67322374Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:03.674308432Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:03.765966136Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:04.678195222Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:04.723712269Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:07.201801579Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:07.256540829Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:07.283227042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:07.284278237Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:07.286172363Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:07.286412355Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:08.019206223Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:08.377897185Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:09.658485024Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:09.658814211Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:09.706762613Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:09.707361127Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:10.011806647Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:10.539857322Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:10:10.860785659Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:12.439458669Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:12.440049422Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:12.517571883Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:14.847685807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:16.85083091Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:16.851322386Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:16.958288816Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:16.958536458Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:17.077384234Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:18.12637991Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:19.409083178Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:19.718174232Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:19.718885403Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:10:20.85946538Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:21.230452519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:21.23163519Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:21.937588837Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:21.937962779Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:22.77275708Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:23.134960464Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:23.135237457Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:25.704239425Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:25.744195433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:25.744403058Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:26.271976567Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:26.272244314Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:27.327133463Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:27.327415225Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:28.173464955Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:28.174172638Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:28.789475831Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:28.79002952Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:28.93725819Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:10:30.859452747Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:31.770218463Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:31.770517877Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:32.287600047Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:32.287987843Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:32.380202665Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:32.380536283Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:32.528091776Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:32.528910082Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:32.602445135Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:32.603469407Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:32.905638927Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:33.21380313Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:33.214258827Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:33.795207177Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:33.795920031Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:33.816025591Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:33.817172635Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:33.819329376Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:34.151055465Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:34.151285373Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:35.309911343Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:35.576886533Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:35.577352709Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:36.60952749Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:36.610027425Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:36.925953709Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:37.170312196Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:37.170778845Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:39.320539831Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:39.321039499Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:40.700578315Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:40.700829707Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:10:40.859435096Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:41.254414626Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:41.254783698Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:41.493635047Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:41.527313613Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:41.527639522Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:41.55409604Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:41.554353747Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:41.715110739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:42.549980938Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:42.550447226Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:43.913455524Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:44.001619257Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:44.274341871Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:45.13741148Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:45.137863267Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:45.358105332Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:45.358866801Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:45.757071663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:45.757774182Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:45.92695952Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:45.927612581Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:46.37507854Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:46.598033628Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:46.59837659Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:46.985955818Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:47.061565136Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:47.061904239Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:48.261135689Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:48.612570745Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:48.702147712Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:48.70262353Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:49.235381682Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:49.493573339Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:49.494488195Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:49.650822016Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:10:50.859413069Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:51.709233347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:51.710010887Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:51.913945983Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:51.914607104Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:52.069904958Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:52.070322747Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:52.082079646Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:52.57841055Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:52.692380802Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:52.999586053Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:53.000087371Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:53.326275447Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:53.907033023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:54.98737213Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:54.987600016Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:55.378532956Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:55.379252095Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:55.413617032Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:55.413812466Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:55.891140036Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:55.89174604Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:56.402796148Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:58.254624751Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:58.854901651Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:59.099252813Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:59.099513906Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:59.713130839Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:59.713393579Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:59.88647892Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:10:59.886607633Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:11:00.859439952Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:00.93297957Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:01.049852668Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:01.050305425Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:01.745284613Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:02.345526126Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:02.345683672Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:02.713554525Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:02.714587688Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:03.698259479Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:04.67733872Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:04.73811327Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:07.175391872Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:07.207564411Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:07.232247339Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:07.232535708Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:07.266941435Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:07.267798922Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:08.010683899Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:08.547721115Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:09.406423139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:09.40678696Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:09.696273038Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:09.696722181Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:10.010312511Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:10.43197119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:11:10.859482476Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:12.444282274Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:12.444875012Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:12.595312444Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:14.844573999Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:16.842300587Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:16.84276131Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:17.074047803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:17.471890688Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:17.472288686Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:18.144750619Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:19.408726463Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:19.992473384Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:19.992986645Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:11:20.859434253Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:21.210238422Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:21.21056101Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:21.930937328Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:21.931085029Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:22.758148459Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:23.150250142Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:23.150509296Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:25.69749484Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:25.743925652Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:25.744252489Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:26.278301091Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:26.278586287Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:27.38108683Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:27.381382057Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:28.168577986Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:28.169295362Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:28.768446876Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:28.769098803Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:28.934206119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:11:30.859433183Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:31.772681272Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:31.772952924Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:32.23879973Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:32.239203768Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:32.345596404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:32.345983461Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:32.458753214Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:32.459291975Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:32.593087185Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:32.593520212Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:32.905523215Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:33.215166252Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:33.215483358Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:33.795323814Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:33.79607061Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:33.816262338Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:33.817348263Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:33.81948737Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:34.156484708Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:34.156854229Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:35.308582069Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:35.576835211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:35.577275882Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:36.634368244Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:36.634949072Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:36.920595952Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:37.171062493Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:37.171504842Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:39.150828112Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:39.151294276Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:40.685841658Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:40.686108891Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:11:40.859488901Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:41.244536969Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:41.244917633Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:41.49104236Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:41.530780224Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:41.531174917Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:41.56409978Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:41.564450544Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:41.717676267Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:42.213092599Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:42.21350667Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:43.861125883Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:43.998280317Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:44.271259683Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:45.137861532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:45.138163727Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:45.358096947Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:45.358905929Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:45.726083363Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:45.727017043Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:45.920080671Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:45.920642125Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:46.375420049Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:46.672031258Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:46.672388422Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:46.981259988Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:47.0611884Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:47.06150397Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:48.258086805Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:48.554582674Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:48.641943776Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:48.64252427Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:49.200972665Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:49.555422962Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:49.555629268Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:49.635698393Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:11:50.859416317Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:51.703525478Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:51.704356077Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:51.913632468Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:51.914301864Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:51.985484766Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:52.068535868Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:52.069032375Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:52.577006171Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:52.693528724Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:52.998894693Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:52.999318707Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:53.329786931Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:53.896866739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:54.972929183Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:54.973113094Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:55.296146839Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:55.296622091Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:55.470914892Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:55.471076246Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:55.645258506Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:55.645706531Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:56.402514015Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:58.255252651Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:58.852092339Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:59.274285336Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:59.274602749Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:59.714351939Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:59.71463927Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:59.899583852Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:11:59.899759308Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:12:00.859432347Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:00.930291389Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:00.956638672Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:00.957125414Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:02.371372736Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:02.373910734Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:02.451247163Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:02.794337885Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:02.795125613Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:03.698152814Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:04.674534959Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:04.722522065Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:07.181356147Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:07.217272089Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:07.236930256Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:07.237124134Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:07.291614164Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:07.292428855Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:08.010113575Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:08.639256649Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:09.442163351Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:09.442899839Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:09.696403296Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:09.700963285Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:10.04765861Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:10.43406963Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:12:10.859455075Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:12.438456561Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:12.439078597Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:12.596760707Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:14.840894341Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:16.716178834Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:16.716624744Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:17.076810832Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:17.418254853Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:17.418543866Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:18.165450269Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:19.409578343Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:19.694307199Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:19.694902769Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:12:20.859451094Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:21.212392967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:21.212812088Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:21.940198347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:21.940368599Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:22.77470198Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:23.135081815Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:23.135395473Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:25.698827443Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:25.744204049Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:25.744444231Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:26.27194477Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:26.272223829Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:27.38251845Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:27.382792073Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:28.245782006Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:28.246410612Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:28.756786164Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:28.757312865Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:28.936806107Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:12:30.859430847Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:31.648019113Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:31.648649055Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:31.788339726Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:31.788586404Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:31.963794517Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:31.964178319Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:32.485739075Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:32.486119924Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:32.593203359Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:32.593658757Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:32.906305698Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:33.21384232Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:33.214153844Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:33.795381738Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:33.796134018Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:33.815888177Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:33.817049578Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:33.819219637Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:34.151855984Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:34.152102029Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:35.310996218Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:35.643540393Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:35.644031608Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:36.616746057Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:36.617156634Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:36.921923113Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:37.16102313Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:37.161454324Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:39.181308176Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:39.182042906Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:40.687561407Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:40.687822758Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:12:40.859454321Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:41.244691915Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:41.245002535Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:41.440065436Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:41.440391003Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:41.488337585Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:41.561186547Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:41.561644596Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:41.717686069Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:42.547460405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:42.547922646Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:43.876968496Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:43.998053507Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:44.341445318Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:45.183166141Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:45.183958713Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:45.367752632Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:45.369421588Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:45.734391249Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:45.735907209Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:45.952791019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:45.954076486Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:46.378155931Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:46.655319303Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:46.655630939Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:46.982344758Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:47.059793073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:47.060099669Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:48.267204281Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:48.550455867Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:48.551141479Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:48.554293038Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:49.200777384Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:49.554805143Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:49.555030562Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:49.646353328Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:12:50.859431594Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:51.716788186Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:51.717756735Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:51.906573138Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:51.907230109Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:51.986392544Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:52.089403289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:52.089861811Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:52.581845439Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:52.691765567Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:52.998884174Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:52.999379204Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:53.352186468Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:53.899329264Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:54.976198274Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:54.976428372Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:55.296189211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:55.296628317Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:55.472440805Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:55.472623436Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:55.718142195Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:55.71858672Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:56.410030949Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:58.298295628Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:58.851770716Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:59.296637849Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:59.297147194Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:59.718019524Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:59.718313208Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:59.870270434Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:12:59.870431191Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:13:00.859437427Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:00.920316805Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:00.920847905Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:00.930802262Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:01.85469734Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:02.368929734Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:02.369091509Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:02.671227925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:02.671932145Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:03.70000616Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:04.680077511Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:04.72056794Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:07.174974827Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:07.20553234Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:07.230882263Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:07.231144426Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:07.255941385Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:07.256692688Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:08.010873177Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:09.306555818Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:09.641361853Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:09.643205694Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:09.701012894Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:09.701613771Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:10.038980622Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:10.435566455Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:13:10.859514065Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:12.440938929Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:12.441591904Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:12.596205343Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:14.841555516Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:16.607587939Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:16.607926554Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:16.729602643Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:16.730190534Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:17.17037167Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:18.132246456Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:19.406657764Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:19.69178593Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:19.692318292Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:13:20.859425605Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:21.212526455Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:21.213072218Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:21.923657238Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:21.923841264Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:22.763537776Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:23.134791256Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:23.135043485Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:25.698261463Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:25.744178805Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:25.744418566Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:26.27270979Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:26.273021858Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:27.3268762Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:27.327160856Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:28.16097144Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:28.16156613Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:28.531646068Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:28.532096064Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:28.937629785Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:13:30.859429861Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:31.626075095Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:31.626687887Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:31.788325573Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:31.788606182Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:31.980760719Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:31.981133327Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:32.456331024Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:32.456806585Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:32.595890268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:32.596349123Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:32.905719232Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:33.214791714Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:33.215108156Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:33.795456548Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:33.796208894Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:33.816090747Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:33.817233289Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:33.819574539Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:34.166966389Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:34.167208962Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:35.310827338Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:35.576731028Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:35.577189472Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:36.646353427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:36.647183497Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:36.948783079Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:37.160968741Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:37.161401397Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:39.110704345Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:39.111143947Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:40.68581714Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:40.686029Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:13:40.859446667Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:41.248164709Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:41.248451196Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:41.415725126Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:41.416034554Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:41.487620936Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:41.557794003Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:41.558072428Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:41.717171974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:42.542586082Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:42.543034821Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:43.857583559Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:43.997371547Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:44.338565844Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:44.548632103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:44.548932797Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:45.358116696Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:45.358872334Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:45.714560685Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:45.715205046Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:45.924650412Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:45.925277157Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:46.380403225Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:46.592909241Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:46.593263275Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:46.985123348Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:47.076643456Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:47.076980507Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:48.2567938Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:48.511212746Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:48.511650828Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:48.553196316Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:49.201051364Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:49.55707222Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:49.557288434Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:49.654959556Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:13:50.859418289Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:51.710379478Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:51.711132295Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:51.91026215Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:51.910930256Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:51.984602846Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:52.073918699Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:52.074332155Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:52.576931258Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:52.692043337Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:53.013043513Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:53.013495648Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:53.302067716Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:53.900140391Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:54.973914984Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:54.974141291Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:55.331909364Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:55.33240747Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:55.476944343Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:55.477098015Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:55.757343169Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:55.757807041Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:56.402437156Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:58.299312459Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:58.852038899Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:59.275524697Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:59.275818554Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:59.710962012Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:59.711240014Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:59.871032132Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:13:59.871184525Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:14:00.859442582Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:00.931627197Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:01.00221445Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:01.002772709Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:01.717029377Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:02.377488019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:02.377611446Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:02.681148435Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:02.681794199Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:03.698489836Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:04.686098304Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:04.720937555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:07.182834134Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:07.225804186Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:07.252844063Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:07.253218574Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:07.270092695Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:07.27089907Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:08.018954514Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:09.428266663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:09.430674972Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:09.535835136Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:09.699603619Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:09.700920146Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:10.022347748Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:10.431848553Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:14:10.859463753Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:12.445784496Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:12.446365108Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:12.517493497Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:14.851575473Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:16.724799768Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:16.725223782Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:17.04480476Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:17.446189027Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:17.446560792Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:18.261957224Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:19.409396517Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:19.99351924Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:19.994047719Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:14:20.859439527Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:21.218700799Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:21.219014996Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:21.902820488Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:21.902991268Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:22.773452522Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:23.134620083Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:23.134899444Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:25.6974206Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:25.744365433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:25.744601817Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:26.272062331Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:26.27229848Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:27.381625829Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:27.381908296Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:28.166818284Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:28.167300753Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:28.606539694Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:28.607110144Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:28.941524523Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:14:30.859429595Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:31.667755285Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:31.668486722Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:31.787514491Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:31.787856293Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:32.292066516Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:32.292527059Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:32.464020507Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:32.464350887Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:32.592548923Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:32.592993695Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:32.905525654Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:33.213691678Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:33.214013038Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:33.795281175Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:33.796090052Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:33.817295812Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:33.818456145Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:33.820582206Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:34.150562654Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:34.150855219Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:35.30889864Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:35.577353341Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:35.57779346Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:36.660902066Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:36.66139419Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:36.947621798Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:37.174932801Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:37.175359978Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:39.311404408Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:39.312031164Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:40.687122004Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:40.687372642Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:14:40.859427985Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:41.047042415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:41.047335352Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:41.487495086Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:41.507457941Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:41.507822892Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:41.557326488Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:41.557609926Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:41.715889664Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:42.552047375Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:42.552560532Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:43.862083756Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:43.997531584Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:44.339277047Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:45.129636042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:45.129941498Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:45.358206425Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:45.359085079Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:45.723035043Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:45.724062241Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:45.914831908Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:45.915448029Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:46.38326375Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:46.705881638Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:46.707754691Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:46.986938531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:47.11345159Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:47.114218484Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:48.258097394Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:48.512718431Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:48.513163473Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:48.558579529Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:49.210107191Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:49.556349546Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:49.556587489Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:49.642454516Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:14:50.859434757Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:51.705760218Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:51.706418311Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:51.91833982Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:51.91891829Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:51.984644743Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:52.070924442Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:52.071334029Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:52.578423079Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:52.691059612Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:52.999123449Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:52.999549425Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:53.300977142Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:53.898122179Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:54.990436352Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:54.990647494Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:55.332808124Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:55.333223678Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:55.418955019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:55.419121061Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:55.746433964Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:55.746935415Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:56.412547779Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:58.307142507Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:58.854361739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:59.287442332Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:59.287738121Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:59.711826152Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:14:59.712178254Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:00.061087614Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:00.061253656Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:15:00.859485246Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:00.930252658Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:01.04019052Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:01.040654322Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:02.636495131Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:02.64082887Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:03.752365179Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:03.807292885Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:04.264259416Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:04.264941924Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:04.68622436Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:04.72825281Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:07.169090914Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:07.199481504Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:07.218286401Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:07.218528479Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:07.250290267Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:07.25107182Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:08.012652092Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:08.543406985Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:09.024289313Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:09.02473298Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:09.68184415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:09.682259951Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:10.023376871Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:10.43553501Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:15:10.859500012Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:12.460239586Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:12.460955542Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:12.516844467Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:14.843745731Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:16.841392022Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:16.84200481Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:17.082509879Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:17.451439657Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:17.451826103Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:18.402168498Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:19.407314691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:19.692927424Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:19.693446464Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:15:20.859453454Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:21.207693722Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:21.208009298Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:21.902993202Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:21.903162612Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:22.760596178Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:23.134856496Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:23.135128191Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:25.700745116Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:25.744570479Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:25.744903338Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:26.271780539Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:26.272021408Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:27.384921197Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:27.38518263Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:28.16051089Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:28.161437377Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:28.558706902Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:28.559251855Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:28.939339795Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:15:30.859427759Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:31.629828563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:31.63033887Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:31.786586201Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:31.786852257Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:32.281600232Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:32.282011207Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:32.467063996Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:32.467417814Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:32.593018274Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:32.593487627Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:32.905740776Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:33.231272501Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:33.231635611Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:33.795309294Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:33.796119481Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:33.819604585Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:33.820957942Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:33.823510144Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:34.159936431Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:34.160209031Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:35.317334587Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:35.577543677Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:35.578216178Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:36.672496465Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:36.673309719Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:36.921896638Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:37.161096613Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:37.161561458Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:39.22794113Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:39.228433225Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:40.68580561Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:40.686025296Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:15:40.859442694Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:41.246609738Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:41.246982226Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:41.487388714Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:41.522567397Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:41.522913967Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:41.557044083Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:41.557365343Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:41.707523636Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:42.401378834Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:42.401803851Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:43.953140998Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:43.997454508Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:44.270756817Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:44.54893645Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:44.54925446Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:45.358090279Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:45.35887667Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:45.731128046Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:45.731805754Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:45.914934165Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:45.915533542Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:46.374863382Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:46.773478641Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:46.773895104Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:46.982849257Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:47.082729618Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:47.083123709Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:48.255479671Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:48.553229769Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:48.649725977Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:48.650291766Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:49.203625608Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:49.505334826Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:49.505575245Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:49.646410224Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:15:50.859425919Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:51.719886921Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:51.720708272Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:51.905696733Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:51.906323808Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:51.985680535Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:52.086240791Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:52.086636271Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:52.577705675Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:52.691269582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:52.998802983Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:52.9992187Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:53.314579946Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:53.896842471Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:54.981865726Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:54.982093386Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:55.33290546Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:55.333332882Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:55.412146738Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:55.412302492Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:55.747147271Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:55.747869493Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:56.402651712Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:58.296419622Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:58.854193743Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:59.301589564Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:59.301902744Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:59.714992845Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:59.715323447Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:59.868984237Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:15:59.869124612Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:16:00.859436845Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:00.93100718Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:00.975451373Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:00.975970263Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:01.535905722Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:02.387602415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:02.387794819Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:02.678422831Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:02.67919193Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:03.701094389Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:04.684196092Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:04.729814157Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:07.173738612Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:07.202747842Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:07.229018955Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:07.229280847Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:07.258995613Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:07.259908805Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:08.010990015Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:08.521856424Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:08.813563421Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:08.814083604Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:09.673197648Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:09.673619245Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:10.031370893Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:10.443993511Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:16:10.86189085Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:12.443549674Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:12.444166138Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:12.596781097Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:14.841294386Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:16.841605876Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:16.842072001Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:17.075642146Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:17.421020783Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:17.421323136Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:18.163028025Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:19.408429944Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:19.694879069Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:19.695393118Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:16:20.859422775Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:21.206730204Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:21.207063723Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:21.920645555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:21.92084247Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:22.789753533Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:23.135113021Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:23.135362946Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:25.69923032Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:25.743778977Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:25.744011124Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:26.300055487Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:26.300339659Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:27.38319086Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:27.383482133Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:28.159779402Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:28.16039289Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:28.76529074Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:28.766049683Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:28.954417487Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:16:30.859451207Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:31.621220504Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:31.621782117Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:31.787348624Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:31.787590772Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:32.283318563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:32.28375604Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:32.357762591Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:32.358129954Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:32.59321789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:32.593683388Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:32.905757628Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:33.213979451Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:33.214279008Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:33.795484474Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:33.796202969Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:33.815998167Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:33.817135785Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:33.819322205Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:34.150335606Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:34.150582886Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:35.321019632Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:35.577399558Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:35.577905259Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:36.608813991Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:36.60921739Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:36.924167672Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:37.16085865Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:37.161308445Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:39.344226742Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:39.344976269Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:40.701292276Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:40.701549566Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:16:40.859469336Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:41.242152571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:41.2424748Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:41.47539445Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:41.475736213Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:41.487911922Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:41.558440491Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:41.558788691Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:41.715520455Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:42.557100739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:42.557531303Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:43.951130073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:43.998826155Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:44.272283757Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:45.117738977Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:45.118063549Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:45.358096932Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:45.358834579Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:45.819107165Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:45.819884879Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:45.928491736Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:45.929099709Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:46.374420906Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:46.603299811Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:46.603685723Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:46.985649349Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:47.077804507Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:47.0781471Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:48.554576879Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:48.652255579Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:48.654238021Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:49.257870004Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:49.5364651Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:49.537174807Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:49.640059301Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:50.330328881Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:16:50.859470538Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:51.711612925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:51.712286619Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:51.903015851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:51.903516947Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:51.982848977Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:52.075198914Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:52.075600646Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:52.594106634Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:52.690946728Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:52.998951223Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:52.999388936Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:53.314399196Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:53.896640627Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:54.980388051Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:54.980604183Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:55.33201653Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:55.332465002Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:55.431334308Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:55.431468383Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:55.735744093Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:55.736301681Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:56.402299743Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:58.283439816Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:58.853193332Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:59.094751642Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:59.095066552Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:59.714680302Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:59.714983197Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:59.886841073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:16:59.886981116Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:17:00.85942941Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:00.93062066Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:01.062190656Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:01.0627036Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:01.742592647Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:02.358311034Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:02.358466756Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:02.681359584Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:02.682470814Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:03.697362445Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:04.679267323Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:04.720102119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:07.172632522Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:07.202156865Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:07.222417021Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:07.22264507Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:07.273640753Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:07.274345306Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:08.040829125Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:08.058739773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:09.413407925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:09.413690838Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:09.691412611Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:09.691901249Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:10.013823588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:10.463458781Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:17:10.86074779Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:12.440219626Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:12.440812993Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:12.516290483Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:14.841708787Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:16.724598468Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:16.725058323Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:17.073229782Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:17.451373533Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:17.451705129Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:18.169551247Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:19.410008368Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:19.696191723Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:19.69674257Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:17:20.85945523Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:21.20875441Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:21.209098817Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:21.930454264Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:21.930638325Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:22.759514267Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:23.134685383Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:23.134928894Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:25.698414941Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:25.746271117Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:25.746508121Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:26.2723894Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:26.272792292Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:27.378939191Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:27.379228627Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:28.178150995Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:28.17881352Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:28.699949501Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:28.700464882Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:28.945888305Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:17:30.859429667Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:31.786593744Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:31.786857385Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:32.291391312Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:32.291812734Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:32.352499137Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:32.352863007Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:32.47218964Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:32.472732325Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:32.573117767Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:32.57353827Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:32.9058067Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:33.220013161Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:33.220394336Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:33.795373606Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:33.79618717Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:33.821501441Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:33.822730248Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:33.825794984Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:34.150228438Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:34.15043874Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:35.309648689Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:35.576894371Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:35.57738597Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:36.611843478Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:36.612306536Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:36.919711132Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:37.160876781Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:37.161310667Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:39.193486228Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:39.194050432Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:40.700957055Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:40.701261071Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:17:40.859438275Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:41.285792739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:41.286113482Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:41.487136686Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:41.519441024Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:41.519844472Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:41.580915295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:41.581651246Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:41.712345063Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:42.545169996Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:42.545570485Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:43.859172069Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:43.997538366Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:44.340891331Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:45.11937564Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:45.119719121Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:45.35809883Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:45.358913711Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:45.713879741Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:45.714856628Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:45.922419848Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:45.923022152Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:46.37561053Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:46.6040088Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:46.604345606Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:46.986276936Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:47.081467238Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:47.081812727Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:48.267806752Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:48.554930853Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:48.648300236Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:48.649114725Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:49.201761057Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:49.554000805Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:49.554288949Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:49.643733335Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:17:50.859470666Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:51.716314464Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:51.717061652Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:51.921577612Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:51.92215328Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:51.984714519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:52.069293699Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:52.069725123Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:52.578329259Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:52.691139397Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:52.999633414Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:53.000088791Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:53.302954129Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:53.896943503Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:54.97658074Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:54.976801041Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:55.297141511Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:55.297595874Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:55.47029263Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:55.470455574Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:55.774561058Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:55.774972894Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:56.402380478Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:58.281717687Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:58.851879582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:59.121378821Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:59.121739915Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:59.713144003Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:59.713445679Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:59.897616881Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:17:59.897801588Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:18:00.859726308Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:00.930184302Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:01.05697379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:01.057442906Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:02.345830368Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:02.345990891Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:02.367769682Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:02.728323282Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:02.729060594Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:03.69991258Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:04.679816866Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:04.730232522Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:07.170919226Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:07.20693357Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:07.226280245Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:07.2264974Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:07.29499414Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:07.295764227Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:08.046136066Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:08.514002902Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:09.40922775Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:09.409600773Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:09.691169621Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:09.691624947Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:10.012146857Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:10.430863351Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:18:10.859430039Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:12.463284684Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:12.463915456Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:12.595393378Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:14.843777237Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:16.71517007Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:16.715614831Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:17.041968203Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:17.504779067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:17.505080006Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:18.172012316Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:19.406122391Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:19.687890289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:19.688507907Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:18:20.859433093Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:21.223651445Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:21.224003049Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:21.901261591Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:21.901439045Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:22.767848824Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:23.135444355Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:23.135774578Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:25.696949003Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:25.743755808Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:25.744001036Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:26.271901908Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:26.272178772Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:27.378761825Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:27.379043611Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:28.16019107Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:28.160922769Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:28.708390743Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:28.70901102Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:28.938171753Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:18:30.859425581Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:31.787327958Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:31.787579503Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:31.967498675Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:31.967939452Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:32.364026656Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:32.364461236Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:32.479398341Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:32.479936005Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:32.593094536Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:32.593523299Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:32.905551757Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:33.213976106Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:33.214297742Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:33.795296408Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:33.796030174Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:33.817217606Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:33.81846604Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:33.820810544Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:34.150237215Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:34.150503281Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:35.309202345Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:35.586495026Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:35.586999721Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:36.705077091Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:36.705453849Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:36.921367281Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:37.172292277Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:37.172777659Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:39.428341378Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:39.43062336Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:40.717706662Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:40.717990006Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:18:40.859466263Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:41.272716232Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:41.273053914Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:41.487398357Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:41.520394333Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:41.520764636Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:41.553550024Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:41.553886256Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:41.722058839Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:42.218078518Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:42.218514388Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:43.85687541Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:43.997802214Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:44.271659302Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:44.554643655Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:44.554956468Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:45.358108048Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:45.35889009Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:45.713003204Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:45.713682781Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:45.919756828Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:45.920369778Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:46.37694906Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:46.58930554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:46.589697596Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:46.983241372Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:47.22072106Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:47.221033457Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:48.255550347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:48.553993591Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:48.651287243Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:48.65190364Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:49.201246644Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:49.567264071Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:49.567486659Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:49.668411586Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:18:50.86180672Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:51.744903671Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:51.746460205Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:51.931766701Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:51.933025489Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:51.987108049Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:52.081911543Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:52.08235589Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:52.587048554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:52.690797975Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:52.999911138Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:53.000326643Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:53.31504354Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:53.897057125Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:54.973165378Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:54.973377952Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:55.296040583Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:55.296429074Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:55.482655438Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:55.482854457Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:55.755582895Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:55.755974229Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:56.402887672Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:58.298919573Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:58.852181192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:59.294798512Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:59.295124004Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:59.714620807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:59.714924755Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:59.898037329Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:18:59.898172943Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:19:00.859432941Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:00.930766247Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:01.049875883Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:01.050349158Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:01.538845433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:02.355610654Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:02.355786899Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:02.692552942Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:02.693337999Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:03.701849826Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:04.677255466Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:04.720188665Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:07.184995713Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:07.21640654Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:07.23610471Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:07.236344068Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:07.247300088Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:07.248054063Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:08.012529227Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:08.547889103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:09.478277799Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:09.478898893Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:09.733807376Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:09.734891303Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:10.013502268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:10.431259768Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:19:10.859449962Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:12.442491893Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:12.443058642Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:12.515316857Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:14.910942637Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:16.84077462Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:16.841255725Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:17.078229259Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:17.411347608Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:17.41162831Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:18.26619778Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:19.410412586Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:19.689691581Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:19.690158834Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:19:20.859417651Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:21.207436791Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:21.207778541Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:21.930573818Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:21.930774194Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:22.758367086Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:23.13490804Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:23.135154638Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:25.697987448Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:25.743895417Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:25.744119569Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:26.272061063Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:26.272346446Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:27.381881628Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:27.382167249Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:28.160045467Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:28.160619364Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:28.706767856Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:28.707313736Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:28.934704669Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:19:30.859438536Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:31.647272664Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:31.647893828Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:31.786711398Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:31.786953176Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:31.956273898Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:31.956696021Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:32.361718717Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:32.362076483Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:32.593591326Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:32.594034833Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:32.905704743Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:33.219326562Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:33.219639353Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:33.795502313Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:33.796274766Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:33.816349091Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:33.817481647Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:33.819712606Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:34.152048723Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:34.152363731Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:35.309836224Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:35.577330532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:35.577886801Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:36.605103814Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:36.605570563Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:36.946286872Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:37.168676025Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:37.169079389Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:39.155784424Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:39.156246955Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:40.70088446Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:40.701144835Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:19:40.859439872Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:41.251611202Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:41.25202131Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:41.487270083Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:41.520594979Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:41.521204875Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:41.557856766Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:41.558159113Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:41.72313148Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:42.562110211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:42.562545383Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:43.861054255Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:43.997055847Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:44.338904403Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:45.122022694Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:45.122337136Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:45.358078069Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:45.358865542Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:45.715044126Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:45.715741981Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:45.929123984Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:45.929772723Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:46.374711997Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:46.602792455Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:46.60313812Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:46.980481718Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:47.227092961Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:47.227390358Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:48.285565986Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:48.553605951Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:48.644542345Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:48.645174788Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:49.201858833Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:49.557118089Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:49.557367679Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:49.666546981Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:19:50.859441218Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:51.709427957Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:51.710097411Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:51.901340213Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:51.902015632Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:51.982454464Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:52.068245767Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:52.068601001Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:52.577699593Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:52.692626272Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:52.999138188Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:52.999573441Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:53.301687656Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:53.896853479Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:54.973072359Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:54.97325397Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:55.29613303Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:55.296543903Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:55.47168316Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:55.471852728Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:55.648030424Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:55.64846002Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:56.40233199Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:58.284700629Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:58.851714824Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:59.290647137Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:59.291004724Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:59.71148307Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:59.711796461Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:59.881367498Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:19:59.88152922Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:20:00.859414401Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:00.931466158Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:01.107878916Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:01.108499134Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:02.903012322Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:02.91085765Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:03.974459716Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:04.102515375Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:04.109559075Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:04.448115522Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:04.732268943Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:04.778066031Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:07.18895675Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:07.219084741Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:07.240083748Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:07.240332408Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:07.265267069Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:07.266024834Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:08.010944552Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:09.448687234Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:09.4491269Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:10.025314927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:10.026119069Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:10.126763624Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:10.355098132Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:10.513445057Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:20:10.862533711Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:12.442969999Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:12.443613763Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:12.523159845Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:14.844530554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:16.846189126Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:16.846624597Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:17.077935248Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:17.541576541Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:17.541936325Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:18.190270821Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:19.414018199Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:19.997069222Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:19.997601722Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:20:20.859444973Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:21.224423715Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:21.224766992Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:21.928437945Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:21.928606977Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:22.773833808Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:23.138808115Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:23.139077818Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:25.697759284Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:25.863261015Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:25.863548613Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:26.27220542Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:26.272478993Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:27.311495361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:27.311828195Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:28.19846245Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:28.199184041Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:28.745449444Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:28.746024585Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:28.940869514Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:20:30.859437029Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:31.624367205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:31.624904669Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:31.985974701Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:31.986286818Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:32.280080032Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:32.280465761Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:32.671951989Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:32.672544232Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:32.679548781Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:32.679944213Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:32.905844724Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:33.213872186Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:33.214144861Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:33.795230383Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:33.795958252Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:33.815476098Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:33.816600707Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:33.818812141Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:34.151298564Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:34.151609895Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:35.312386794Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:35.577226704Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:35.577714584Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:36.604527684Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:36.604935129Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:36.925084354Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:37.162183457Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:37.162614269Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:39.309873415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:39.310263502Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:40.687219506Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:40.687479952Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:20:40.8594354Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:41.249612426Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:41.249930539Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:41.48650842Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:41.510026725Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:41.510479149Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:41.563986262Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:41.564331401Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:41.722452585Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:42.557174705Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:42.557621323Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:43.857931927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:44.000053402Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:44.272010827Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:44.562969487Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:44.563300123Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:45.35812121Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:45.358970331Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:45.718492712Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:45.719192357Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:46.017981208Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:46.018547215Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:46.376305571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:46.595925503Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:46.596315125Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:46.981083387Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:47.215129015Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:47.215467889Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:48.254256033Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:48.561112695Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:48.703536211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:48.704253755Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:49.25165175Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:49.558368063Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:49.558659084Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:49.68778303Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:20:50.859458162Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:51.702313504Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:51.703064674Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:51.894772451Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:51.895329266Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:51.989458494Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:52.145260587Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:52.146489967Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:52.578057468Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:52.704752428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:53.001166928Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:53.001994124Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:53.30847586Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:53.901130851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:54.973785575Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:54.974031624Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:55.356996309Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:55.357475949Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:55.472553564Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:55.472890363Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:55.65092728Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:55.651385033Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:56.457688728Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:58.307414261Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:58.8762569Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:59.103594507Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:59.103892953Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:59.714048432Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:59.714310522Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:59.871168332Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:20:59.871359337Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:21:00.859439192Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:00.931546032Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:01.072853608Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:01.073353449Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:01.552352667Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:02.370700619Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:02.370882573Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:02.709897037Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:02.71055108Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:03.795717977Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:04.68308975Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:04.725824738Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:07.174401567Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:07.203763932Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:07.229517013Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:07.229819535Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:07.26502551Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:07.265780863Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:08.012216275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:09.463943534Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:09.464346221Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:09.586184639Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:09.699385445Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:09.69999264Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:10.013032052Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:10.438351709Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:21:10.859447817Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:12.432024024Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:12.43277294Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:12.517466181Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:14.848063198Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:16.740936225Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:16.741415814Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:17.075807172Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:17.440615347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:17.440913134Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:18.200864016Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:19.411863572Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:19.688107076Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:19.688611287Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:21:20.859432772Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:21.211266726Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:21.21164242Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:21.935824693Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:21.935984463Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:22.760084848Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:23.134779639Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:23.135035129Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:25.698605083Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:25.744082722Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:25.744325729Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:26.292900993Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:26.293165737Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:27.380955175Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:27.38122979Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:28.167244831Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:28.173375114Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:28.607649884Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:28.608081566Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:28.941928554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:21:30.859453022Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:31.648796864Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:31.649487867Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:31.786848743Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:31.78708463Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:32.273530382Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:32.273931348Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:32.457783283Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:32.458250802Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:32.593390195Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:32.593932301Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:32.905632843Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:33.21443863Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:33.214783117Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:33.79548901Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:33.796254308Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:33.816636441Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:33.817771955Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:33.820011454Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:34.150737657Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:34.15095062Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:35.3117815Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:35.577079191Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:35.577575278Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:36.617041005Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:36.617449176Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:36.926254355Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:37.165563207Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:37.166014457Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:39.372582269Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:39.37309008Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:40.695601295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:40.695913181Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:21:40.859442134Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:41.250127044Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:41.250455323Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:41.487910254Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:41.519109227Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:41.519474324Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:41.555332564Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:41.555691922Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:41.729814781Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:42.5529556Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:42.553386561Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:43.859472608Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:43.998018534Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:44.337912984Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:45.135052074Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:45.13532494Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:45.358089058Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:45.358837546Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:45.721064512Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:45.72183566Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:45.938526634Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:45.939212598Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:46.398923077Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:46.619427205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:46.619813502Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:46.981602484Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:47.079643335Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:47.079961883Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:48.258368019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:48.553964372Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:48.669556851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:48.670175574Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:49.200401388Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:49.493410899Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:49.493682252Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:49.654943127Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:21:50.859420588Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:51.711791551Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:51.712510054Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:51.903176753Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:51.903743281Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:51.985929871Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:52.079270402Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:52.079752062Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:52.577470707Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:52.69503849Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:52.999701688Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:53.000275881Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:53.301561282Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:53.897122053Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:54.983287272Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:54.983520472Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:55.331926031Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:55.332413656Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:55.470329079Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:55.470495191Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:55.768969304Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:55.769475239Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:56.404544117Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:58.270201829Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:58.915560888Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:59.310970045Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:59.311366055Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:59.714163858Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:21:59.714481279Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:00.051932295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:00.052089372Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:22:00.859434814Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:00.930384017Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:00.975458023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:00.975890415Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:01.515614971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:02.371596415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:02.371782105Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:02.669503576Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:02.670476314Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:03.699984335Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:04.676879694Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:04.724937215Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:07.17130932Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:07.200346201Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:07.226110004Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:07.226359691Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:07.254118557Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:07.254838398Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:08.016196991Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:08.261560706Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:09.696560409Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:09.697056503Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:09.776867742Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:09.777418727Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:10.049241645Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:10.433059683Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:22:10.859434711Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:12.438288272Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:12.438890062Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:12.51669046Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:14.840691812Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:16.715937615Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:16.716495511Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:17.081634322Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:17.414141515Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:17.414508303Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:18.300654444Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:19.408895177Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:19.68738671Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:19.687928959Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:22:20.859423889Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:21.208208916Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:21.20861747Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:21.910819988Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:21.91097763Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:22.759595074Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:23.145385423Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:23.145631199Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:25.698376818Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:25.743761777Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:25.743984884Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:26.271982533Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:26.272246269Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:27.3814982Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:27.381947793Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:28.159048109Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:28.15960879Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:28.619587816Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:28.620150973Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:28.96259983Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:22:30.859432441Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:31.643182035Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:31.643906719Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:31.786980231Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:31.787275948Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:32.300384607Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:32.300877303Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:32.451489015Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:32.452085985Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:32.597373915Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:32.597949057Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:32.905580622Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:33.215303379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:33.215712488Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:33.795322235Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:33.796193487Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:33.817838025Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:33.819119714Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:33.821506561Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:34.150757934Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:34.150979339Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:35.311193366Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:35.577273149Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:35.57775908Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:36.608843681Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:36.609245992Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:36.920444175Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:37.161852853Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:37.162399453Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:39.307732334Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:39.30831451Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:40.685829939Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:40.686094517Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:22:40.859432855Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:41.251167727Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:41.25149787Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:41.49032575Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:41.511529224Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:41.511882454Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:41.561821594Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:41.562175315Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:41.708310482Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:42.572340548Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:42.572887983Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:43.863609726Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:44.000464329Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:44.339361681Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:45.120528628Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:45.120849738Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:45.358052148Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:45.358814748Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:45.721800024Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:45.722932477Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:45.949413775Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:45.950035036Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:46.374864787Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:46.61197569Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:46.612393263Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:46.981473175Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:47.060681324Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:47.060998185Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:48.264992997Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:48.554559243Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:48.641475215Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:48.642045855Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:49.201589278Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:49.504692479Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:49.50496544Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:49.657095825Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:22:50.859452905Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:51.6693609Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:51.670148375Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:51.905512427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:51.906057462Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:51.986959826Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:52.073123416Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:52.073549937Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:52.579251142Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:52.693921738Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:52.99885833Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:52.99929531Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:53.325905481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:53.906959529Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:55.015933832Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:55.016150484Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:55.33819747Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:55.33863564Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:55.429920176Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:55.430079991Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:55.763548981Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:55.763959797Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:56.403582061Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:58.258711191Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:58.861255748Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:59.303513126Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:59.303871273Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:59.729719648Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:22:59.730014871Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:00.05024186Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:00.050398399Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:23:00.859422607Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:00.930602272Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:01.059640416Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:01.060307246Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:01.814082174Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:02.367368485Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:02.367499403Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:02.699797851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:02.700492634Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:03.699872229Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:04.678653336Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:04.719920808Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:07.17051874Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:07.200566162Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:07.220366533Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:07.220647725Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:07.270273119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:07.271115468Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:08.013915853Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:08.065140803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:09.577553219Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:09.587205242Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:09.716502304Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:09.717291324Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:10.090173451Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:10.430593059Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:23:10.859438634Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:12.362698103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:12.363288925Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:12.516598365Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:14.841563549Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:16.605274805Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:16.605678275Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:16.842422652Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:16.84290765Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:17.071378147Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:18.322515677Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:19.409463154Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:19.994241375Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:19.99479863Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:23:20.859418432Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:21.208854029Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:21.209205952Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:21.935833845Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:21.936542224Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:22.772938498Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:23.135480485Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:23.135822547Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:25.697294294Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:25.747521811Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:25.747767235Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:26.27199416Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:26.272228884Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:27.311181231Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:27.311456815Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:28.165439255Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:28.166106822Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:28.580327727Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:28.580957457Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:28.942796691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:23:30.859429097Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:31.622884791Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:31.623417301Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:31.787050435Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:31.787289938Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:32.280983468Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:32.281380788Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:32.360362525Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:32.360729255Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:32.600366998Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:32.600831361Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:32.9055438Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:33.214440229Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:33.214796005Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:33.795251609Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:33.796031169Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:33.815697878Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:33.817198306Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:33.819422188Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:34.150740681Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:34.151008932Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:35.310346087Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:35.58995973Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:35.590424801Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:36.639972459Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:36.640455573Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:36.929961382Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:37.160691472Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:37.161165592Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:39.187487526Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:39.188442802Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:40.68875983Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:40.689075229Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:23:40.85944604Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:41.257015933Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:41.257305104Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:41.487068975Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:41.505950648Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:41.506274466Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:41.569495261Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:41.570039672Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:41.708289338Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:42.544355241Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:42.544816345Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:43.858208461Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:43.999000148Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:44.338016044Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:44.577127731Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:44.577516529Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:45.358120872Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:45.358900039Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:45.716022373Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:45.716648322Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:45.929372579Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:45.929998615Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:46.377560563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:46.687746375Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:46.688116487Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:46.983008122Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:47.074939406Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:47.075337204Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:48.266258165Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:48.52207424Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:48.523039741Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:48.553238642Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:49.209108519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:49.554394473Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:49.554635382Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:49.662577096Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:23:50.859434747Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:51.673362521Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:51.674088871Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:51.909639635Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:51.910325211Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:51.982954526Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:52.069069263Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:52.069472078Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:52.576705861Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:52.690726556Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:52.999233415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:52.999655853Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:53.33719145Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:53.89680916Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:54.975463727Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:54.975734398Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:55.335971814Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:55.336390644Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:55.475613722Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:55.475797177Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:55.776024754Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:55.776484451Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:56.402924166Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:58.260261839Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:58.852020375Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:59.296222027Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:59.29653165Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:59.735287992Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:23:59.735583129Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:00.059844215Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:00.060009048Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:24:00.859479384Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:00.934389998Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:01.028895422Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:01.029554403Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:01.622542433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:02.359448662Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:02.359629711Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:02.68016763Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:02.680898239Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:03.699258939Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:04.681851203Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:04.722839215Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:07.182222293Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:07.220226125Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:07.246620907Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:07.246887885Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:07.258706827Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:07.259408874Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:08.015565452Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:08.892823645Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:08.893236019Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:09.322967555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:09.706512523Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:09.708278592Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:10.022067276Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:10.434253795Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:24:10.85942452Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:12.446236866Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:12.44700084Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:12.517500435Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:14.841104834Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:16.850309758Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:16.850778438Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:16.991039149Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:16.991311062Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:17.079614138Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:18.175979398Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:19.414748218Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:19.684586667Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:19.685106616Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:24:20.859447291Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:21.231959712Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:21.232307881Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:21.928773793Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:21.928956959Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:22.774007881Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:23.134698273Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:23.134940213Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:25.698104453Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:25.744585161Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:25.744816291Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:26.272513215Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:26.272792862Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:27.380680415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:27.380961607Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:28.160502206Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:28.161062839Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:28.816075362Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:28.81659216Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:28.954104522Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:24:30.859421678Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:31.646714759Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:31.647446769Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:31.796727882Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:31.797042532Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:31.97592918Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:31.976419263Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:32.353266157Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:32.353656801Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:32.592945828Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:32.593520785Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:32.908983363Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:33.215384381Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:33.215829491Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:33.795289512Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:33.796058566Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:33.817045572Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:33.818260294Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:33.8205746Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:34.162397065Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:34.162648002Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:35.310933795Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:35.577027865Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:35.577474173Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:36.598648437Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:36.59909877Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:36.921116745Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:37.174431955Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:37.17485732Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:39.271460066Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:39.272023997Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:40.685836866Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:40.686094001Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:24:40.859517039Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:41.048328185Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:41.048631095Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:41.486439493Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:41.513503462Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:41.513897057Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:41.558141512Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:41.558471757Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:41.719784622Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:42.54448101Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:42.544926585Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:43.855648695Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:43.999357519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:44.275391739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:45.124385682Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:45.124695641Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:45.359281624Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:45.360497322Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:45.713520315Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:45.714212856Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:45.927859023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:45.928475081Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:46.376193795Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:46.605179014Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:46.605517089Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:46.985059487Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:47.074982262Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:47.075303028Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:48.257516663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:48.511864902Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:48.512533914Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:48.553387547Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:49.201007778Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:49.55500501Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:49.555278428Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:49.641001005Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:24:50.859454224Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:51.626690007Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:51.627415469Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:51.901741968Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:51.902297337Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:51.984629373Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:52.111536508Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:52.112075465Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:52.577005176Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:52.692861071Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:52.999130546Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:52.999562315Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:53.327277041Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:53.897758715Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:55.030037232Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:55.030525968Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:55.314409707Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:55.315231667Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:55.478333341Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:55.478649445Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:55.942291894Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:55.943426778Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:56.407800497Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:58.257023961Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:58.854898651Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:59.29868663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:59.298999705Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:59.723202922Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:59.723458726Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:59.886267446Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:24:59.886432904Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:25:00.860180685Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:00.930283065Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:01.034384101Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:01.03489645Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:01.616113955Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:02.71114332Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:02.741952538Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:03.307103073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:03.308159165Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:03.697529633Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:04.681578589Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:04.719956924Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:07.17310659Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:07.202256725Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:07.221043108Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:07.22125487Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:07.276084935Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:07.276809626Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:08.033958198Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:08.18924199Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:08.880253746Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:08.880640312Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:09.707834486Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:09.708360989Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:10.012858363Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:10.430939845Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:25:10.8594289Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:12.470549118Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:12.471541728Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:12.595760717Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:14.857252911Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:16.854945656Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:16.856434317Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:17.07465057Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:17.350168497Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:17.350469937Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:18.17094467Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:19.407005865Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:19.689775331Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:19.690262222Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:25:20.859434756Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:21.228245479Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:21.228606549Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:21.868349127Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:21.86852948Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:22.758392016Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:23.134862508Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:23.135107512Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:25.700161044Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:25.744506887Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:25.744844049Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:26.273119874Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:26.273462484Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:28.161213361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:28.161905001Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:28.751590945Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:28.752051291Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:28.932867825Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:25:30.860138325Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:31.626754982Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:31.627344339Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:31.787401379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:31.787678251Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:31.963271599Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:31.963687947Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:32.356464696Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:32.35682941Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:32.592925101Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:32.59332141Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:32.905573993Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:33.214070661Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:33.214373161Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:33.795269312Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:33.79599388Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:33.816599923Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:33.817761751Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:33.820001247Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:34.150437568Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:34.150725028Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:35.310136035Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:35.576610661Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:35.577076604Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:36.604962412Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:36.605302626Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:36.920340773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:37.161306061Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:37.16178159Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:37.302738709Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:37.303001714Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:39.245197254Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:39.245809435Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:40.701024298Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:40.701304257Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:25:40.85942972Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:41.243135236Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:41.24352583Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:41.486141788Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:41.508760515Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:41.509179425Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:41.555039946Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:41.555317177Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:41.711114767Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:42.214191983Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:42.214610405Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:43.857442559Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:43.997752217Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:44.273132527Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:44.551455167Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:44.551792654Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:45.358082428Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:45.358842654Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:45.724095597Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:45.724973383Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:45.929295399Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:45.92998438Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:46.376337004Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:46.783147955Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:46.783486292Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:46.98333719Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:47.071937484Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:47.072259481Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:48.323359347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:48.53399158Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:48.534413434Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:48.556349242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:49.201593028Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:49.564611062Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:49.564892116Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:49.651463666Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:25:50.859432616Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:51.639604589Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:51.642695895Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:51.918902695Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:51.919539261Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:51.983044244Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:52.089564394Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:52.090070593Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:52.576946941Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:52.691140494Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:52.999045225Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:52.999583993Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:53.327599725Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:53.897182042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:54.971408212Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:54.971615604Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:55.296039417Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:55.296491053Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:55.474884678Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:55.475045672Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:55.730415639Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:55.730893545Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:56.411954945Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:58.255779795Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:58.851641739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:59.318135816Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:59.318422018Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:59.723492979Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:59.723820159Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:59.889147147Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:25:59.889300537Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:26:00.859445171Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:00.930792777Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:01.189395696Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:01.189932885Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:02.346313982Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:02.346489644Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:02.416557772Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:02.705912685Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:02.706642522Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:03.699926254Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:04.67778607Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:04.720225526Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:07.173336146Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:07.202800363Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:07.221291505Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:07.221485667Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:07.258138686Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:07.258935839Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:08.020276992Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:08.765913333Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:09.544952131Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:09.545558717Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:09.705279099Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:09.706049643Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:10.050325428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:10.431069659Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:26:10.859457136Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:12.357355498Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:12.358241713Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:12.51990679Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:14.841168401Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:16.724537423Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:16.725500751Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:17.072539157Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:17.42719501Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:17.427588188Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:18.204563769Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:19.40798061Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:19.692756582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:19.693277351Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:26:20.859429398Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:21.208362887Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:21.208698992Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:21.94315914Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:21.94333563Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:22.773688723Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:23.134576396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:23.134874129Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:25.698087094Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:25.744458583Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:25.744720296Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:26.298938424Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:26.29923717Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:27.389519899Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:27.389817627Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:28.163622181Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:28.164227503Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:28.756101942Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:28.75661361Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:28.995756491Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:26:30.85945212Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:31.787164838Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:31.787438782Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:32.183510345Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:32.184003338Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:32.279421561Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:32.279871224Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:32.478177106Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:32.478529347Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:32.599626307Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:32.600089752Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:32.905782865Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:33.224247496Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:33.224548227Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:33.795390893Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:33.796133905Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:33.818981365Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:33.820109402Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:33.822521321Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:34.150540548Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:34.150783191Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:35.310014048Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:35.577470062Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:35.57794859Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:36.602729765Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:36.603129645Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:36.925446985Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:37.165782905Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:37.166223407Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:39.265937521Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:39.266275681Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:40.695013037Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:40.695329978Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:26:40.870762993Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:41.257203192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:41.257520604Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:41.423905305Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:41.424227589Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:41.486483273Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:41.554572272Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:41.554865337Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:41.717794416Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:42.539432275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:42.539889309Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:43.956463525Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:43.998137189Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:44.272519519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:45.136653695Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:45.136988661Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:45.358033128Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:45.358726791Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:45.714890033Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:45.715591688Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:45.92781388Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:45.928406183Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:46.376246168Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:46.602750845Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:46.603087868Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:46.982433927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:47.078451657Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:47.078789958Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:48.257341844Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:48.554897457Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:48.64815602Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:48.648845027Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:49.201615084Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:49.554997213Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:49.555249081Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:49.658574139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:26:50.859419868Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:51.627386405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:51.628105087Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:51.909561574Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:51.91024199Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:51.981791203Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:52.213535861Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:52.213948712Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:52.576995663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:52.690851056Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:52.999069055Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:52.999555157Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:53.331282727Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:53.897603128Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:54.972186175Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:54.972395647Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:55.295928948Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:55.296343036Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:55.47046626Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:55.470651919Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:55.639532619Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:55.639914315Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:56.413246763Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:58.25582805Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:58.857690821Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:59.111697268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:59.112098778Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:59.723345107Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:59.723631352Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:59.886025477Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:26:59.886163216Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:27:00.859428535Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:00.930825839Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:01.055428416Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:01.055908404Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:01.603016265Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:02.377391563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:02.377544239Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:02.681234262Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:02.682177767Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:03.706035583Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:04.674539071Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:04.72059296Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:07.169827532Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:07.198974156Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:07.2237986Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:07.224073124Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:07.251527625Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:07.252239127Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:08.024576619Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:08.075400242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:09.49395271Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:09.494365273Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:09.708367044Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:09.70908692Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:10.0122341Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:10.431376318Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:27:10.859462292Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:12.356989365Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:12.357548298Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:12.516598344Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:14.840874847Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:16.714710454Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:16.715159208Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:17.079798892Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:17.409975116Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:17.410264616Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:18.181176968Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:19.40984014Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:19.680453275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:19.680997388Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:27:20.859422805Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:21.221163479Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:21.221512885Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:21.950188792Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:21.950363906Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:22.773295405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:23.134739647Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:23.135000263Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:25.698473351Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:25.744263983Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:25.744529922Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:26.275803097Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:26.276089719Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:27.314191111Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:27.31445207Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:28.198468151Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:28.198976198Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:28.877447548Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:28.87823234Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:28.931519268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:27:30.859432023Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:31.78740222Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:31.787697478Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:32.289603446Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:32.290026937Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:32.442288615Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:32.442690458Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:32.53611474Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:32.536636423Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:32.596808327Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:32.597240181Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:32.905553275Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:33.213992317Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:33.214330877Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:33.795655186Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:33.796538683Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:33.820290975Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:33.821512714Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:33.824424322Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:34.150167598Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:34.150421154Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:35.314405722Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:35.576652509Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:35.577089209Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:36.618203297Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:36.618632235Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:36.921228632Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:37.16090887Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:37.161365258Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:39.213198527Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:39.213656836Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:40.693158656Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:40.69340825Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:27:40.859419477Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:41.245496255Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:41.245844702Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:41.447329108Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:41.447678181Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:41.486498081Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:41.554651409Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:41.555151831Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:41.721941104Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:42.554687373Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:42.555097405Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:43.859978167Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:43.997876756Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:44.29131036Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:45.136729254Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:45.137020707Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:45.358114746Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:45.358841856Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:45.72439425Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:45.725077465Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:45.929222532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:45.929998027Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:46.37884363Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:46.615751424Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:46.616053479Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:46.986185685Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:47.073929688Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:47.074330983Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:48.266744804Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:48.553962876Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:48.64414634Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:48.644929728Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:49.200422555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:49.557002457Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:49.557249382Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:49.641282047Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:27:50.859426428Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:51.638252584Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:51.639188928Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:51.900914061Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:51.901446455Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:51.984429287Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:52.068609815Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:52.069042534Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:52.582198894Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:52.690722223Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:52.999113065Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:52.999580869Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:53.327789979Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:53.896701811Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:54.992412839Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:54.992711234Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:55.296851823Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:55.297444741Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:55.471209817Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:55.471376006Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:55.660007534Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:55.660440186Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:56.402166549Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:58.270862077Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:58.854298467Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:59.100247168Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:59.100549991Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:59.722944978Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:59.723220111Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:59.886869512Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:27:59.887031619Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:28:00.859429889Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:00.931125119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:00.939191382Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:00.939604144Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:01.718872064Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:02.36920264Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:02.369362816Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:02.669647763Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:02.670561404Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:03.698530793Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:04.678899989Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:04.720567286Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:07.180754804Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:07.210647687Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:07.229924854Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:07.230180418Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:07.284923085Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:07.285785052Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:08.011017581Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:08.457221572Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:09.439517314Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:09.440023606Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:09.698387744Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:09.699248134Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:10.012221761Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:10.430991946Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:28:10.859471024Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:12.370000456Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:12.370801521Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:12.51778895Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:14.861874591Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:16.841861212Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:16.842328859Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:17.041980222Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:17.438956858Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:17.439272661Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:18.156861981Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:19.405229604Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:19.683457718Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:19.68399361Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:28:20.859434959Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:21.221156634Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:21.221524591Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:21.930553093Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:21.930775821Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:22.761349316Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:23.134794623Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:23.135055058Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:25.698697508Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:25.746972537Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:25.747251272Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:26.272002551Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:26.272245786Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:27.38212236Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:27.382394089Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:28.161297242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:28.162979932Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:28.582407392Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:28.582868498Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:28.939528879Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:28:30.859419904Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:31.788613555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:31.788879426Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:32.281539668Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:32.282067617Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:32.462348765Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:32.462825107Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:32.505493789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:32.505968441Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:32.593944472Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:32.59437966Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:32.905513945Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:33.215557263Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:33.215916883Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:33.795366976Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:33.796089254Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:33.81698312Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:33.818155893Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:33.820434105Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:34.151086926Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:34.151339095Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:35.32543828Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:35.578053505Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:35.57867741Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:36.607604063Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:36.607962527Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:36.920968518Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:37.161869561Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:37.162452727Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:39.226425564Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:39.22686062Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:40.700808973Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:40.701061531Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:28:40.859436733Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:41.245363376Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:41.245716737Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:41.487073969Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:41.516974726Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:41.51732615Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:41.554914552Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:41.55518932Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:41.719048477Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:42.542415499Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:42.542883662Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:43.864932361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:44.002887709Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:44.272306566Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:44.549061089Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:44.549376175Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:45.358099836Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:45.358909398Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:45.727899319Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:45.728543044Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:45.928401911Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:45.929188018Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:46.381061548Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:46.596348406Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:46.596650729Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:46.982882983Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:47.062551532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:47.062858653Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:48.278471762Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:48.569973762Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:48.648812872Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:48.649253935Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:49.201195944Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:49.49305953Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:49.493300478Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:49.638025907Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:28:50.859492788Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:51.615896632Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:51.616540026Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:51.894030754Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:51.894565028Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:51.982164278Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:52.068453561Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:52.068883197Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:52.578081814Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:52.692581655Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:52.998623097Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:52.999095635Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:53.332131857Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:53.896616975Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:54.972454642Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:54.972710814Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:55.3325546Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:55.333042338Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:55.470648144Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:55.470820819Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:55.779493821Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:55.779996477Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:56.402480155Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:58.262799148Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:58.855402391Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:59.363474697Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:59.364307925Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:59.731867168Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:59.732370824Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:59.88272833Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:28:59.883021629Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:29:00.859435422Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:00.863703764Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:00.864192834Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:00.931159852Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:01.525302539Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:02.36771374Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:02.367872454Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:02.670821274Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:02.671695243Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:03.697375209Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:04.681962575Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:04.719960119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:07.180107142Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:07.213473376Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:07.239419107Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:07.239714846Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:07.258404841Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:07.259161085Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:08.013499925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:08.995835443Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:09.448332926Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:09.450298418Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:09.700530238Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:09.704827558Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:10.039272681Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:10.432866593Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:29:10.859429494Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:12.355935668Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:12.356541774Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:12.596469864Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:14.918858845Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:16.841384982Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:16.84196319Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:17.172730832Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:17.46801883Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:17.468362474Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:18.238277763Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:19.407406837Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:20.008288067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:20.008806433Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:29:20.859386918Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:21.227430236Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:21.227740019Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:21.925728667Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:21.925907965Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:22.759101417Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:23.134473786Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:23.134780536Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:25.696842064Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:25.744498193Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:25.744785682Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:26.272000522Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:26.27229813Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:27.379284808Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:27.379733549Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:28.165466916Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:28.166162979Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:28.66451774Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:28.66520191Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:28.937913155Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:29:30.859427369Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:31.663503498Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:31.664053805Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:31.787045516Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:31.787313518Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:32.289447524Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:32.289953307Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:32.46387071Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:32.464300515Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:32.594212088Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:32.594648959Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:32.90550551Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:33.21402156Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:33.21436576Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:33.795275479Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:33.796030432Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:33.81649927Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:33.817634744Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:33.819910822Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:34.150511349Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:34.150745294Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:35.309697568Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:35.57679257Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:35.577246472Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:36.678892717Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:36.689254583Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:36.95600562Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:37.207538738Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:37.207994084Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:39.450947233Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:39.451354076Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:40.687318726Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:40.687567441Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:29:40.85943316Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:41.260378511Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:41.260726947Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:41.487143271Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:41.526475058Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:41.526883788Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:41.593958159Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:41.594215711Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:41.715897462Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:42.549094341Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:42.549514256Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:43.864496396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:43.997727303Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:44.339329361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:45.129356759Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:45.129689392Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:45.358341157Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:45.359123479Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:45.71485235Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:45.715535445Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:45.929936672Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:45.93054949Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:46.374935551Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:46.60288867Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:46.603237495Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:46.98386901Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:47.063254951Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:47.063696367Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:48.257188637Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:48.554392571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:48.64106523Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:48.641652774Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:49.202471138Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:49.497259662Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:49.497511141Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:49.635089411Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:29:50.859423122Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:51.631368751Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:51.632075133Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:51.893906932Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:51.894446928Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:51.983083347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:52.084989127Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:52.085418138Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:52.577049225Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:52.691407812Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:53.003397181Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:53.00385251Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:53.329111211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:53.89717506Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:54.988546584Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:54.988803048Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:55.333279462Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:55.333717874Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:55.470787999Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:55.470963142Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:55.788762593Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:55.789197485Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:56.402824857Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:58.265176399Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:58.852045687Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:59.294048967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:59.294415718Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:59.726379667Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:59.726642879Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:59.869872836Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:29:59.870023794Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:30:00.859433453Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:00.930325363Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:01.063643987Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:01.064192988Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:03.681374124Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:03.7531689Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:03.884853617Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:04.40792584Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:04.447182628Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:04.797301133Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:04.810916285Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:04.855903819Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:07.187297652Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:07.221162146Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:07.24198175Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:07.242212776Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:07.286610378Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:07.287765971Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:08.017157058Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:09.545941503Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:09.546382563Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:09.818028041Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:09.818606124Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:10.013751411Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:10.460072044Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:30:10.863814319Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:11.784226165Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:12.517579576Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:12.548040236Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:12.548616967Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:14.842791344Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:16.719286388Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:16.720055338Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:17.005959407Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:17.006358391Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:17.073211839Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:18.556052463Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:19.408189507Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:19.690228287Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:19.690775893Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:30:20.859451707Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:21.211504278Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:21.211906365Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:21.920389279Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:21.920562418Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:22.772959856Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:23.134587754Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:23.134858765Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:25.719056437Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:25.744306513Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:25.744547916Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:26.275363975Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:26.275609659Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:27.311233159Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:27.31150257Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:28.161369623Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:28.162561051Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:28.739425754Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:28.739961542Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:28.963497897Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:30:30.859435568Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:31.620823136Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:31.621380967Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:31.78688792Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:31.78715857Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:31.957808237Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:31.958178862Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:32.359632671Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:32.360014389Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:32.592857214Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:32.593354414Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:32.905570024Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:33.222557005Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:33.222910078Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:33.795299504Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:33.796096502Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:33.817275768Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:33.818545567Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:33.82090501Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:34.150979648Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:34.15131486Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:35.310203486Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:35.586184842Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:35.586677447Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:36.626102569Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:36.626530076Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:37.16098049Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:37.161432912Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:37.470693619Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:39.375514666Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:39.376318169Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:40.686464442Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:40.686779106Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:30:40.862838171Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:41.245791141Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:41.246125908Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:41.490203013Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:41.512765906Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:41.513157234Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:41.570450628Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:41.57089907Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:41.714541384Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:42.555474425Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:42.555938724Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:43.871178129Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:44.001706428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:44.337485682Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:44.550160906Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:44.550456954Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:45.358191315Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:45.359259798Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:45.715874037Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:45.717018704Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:45.928260491Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:45.928886452Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:46.379408548Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:46.644228249Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:46.644715734Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:46.987301135Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:47.078368624Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:47.078808885Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:48.262080818Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:48.558909777Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:48.642554668Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:48.643110727Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:49.201644723Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:49.556304198Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:49.557251684Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:49.670686961Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:30:50.859357488Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:51.638123751Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:51.6392367Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:51.920579797Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:51.921285611Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:51.988304034Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:52.066797666Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:52.067178434Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:52.578797692Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:52.693831477Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:52.999722568Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:53.000276256Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:53.303423054Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:53.896783232Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:55.225502653Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:55.225759623Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:55.339080913Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:55.339518284Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:55.473917033Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:55.474104573Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:55.788383951Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:55.788989188Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:56.404449199Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:58.336765229Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:58.851909628Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:59.112165538Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:59.113056715Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:59.72449292Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:59.724929278Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:59.920811719Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:30:59.921098405Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:31:00.861180034Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:00.957473136Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:01.087910991Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:01.089086976Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:01.948760251Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:02.466956437Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:02.467266609Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:02.679476351Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:02.680113631Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:03.697776279Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:04.72161242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:04.773853943Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:07.176645288Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:07.206236826Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:07.22480823Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:07.225051139Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:08.019001255Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:08.432918113Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:08.433963004Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:09.54334144Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:09.54413855Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:09.72784882Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:09.72844478Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:10.015614249Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:10.432656911Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:31:10.859355101Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:12.353734247Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:12.354329406Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:12.526345034Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:14.314464484Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:14.845867311Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:16.714745736Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:16.715285343Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:17.077678326Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:17.416525545Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:17.416850241Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:18.214266887Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:19.415697371Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:19.990573515Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:19.991092288Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:31:20.859441516Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:21.224161017Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:21.224590014Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:21.928271691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:21.928416363Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:22.760062748Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:23.135240353Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:23.135484746Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:25.698640257Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:25.785687081Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:25.785940033Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:26.344210337Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:26.344488151Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:28.182738384Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:28.183337866Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:28.688794188Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:28.689353291Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:28.942031286Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:31:30.862142742Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:31.616551971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:31.618061976Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:31.669986301Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:31.670592431Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:31.985382551Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:31.986113592Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:32.057256531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:32.057524901Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:32.347204104Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:32.347596276Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:32.703470686Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:32.703943594Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:32.90556593Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:33.214237755Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:33.214606485Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:33.795501858Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:33.796466256Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:33.816568585Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:33.81774411Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:33.819894379Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:34.150176812Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:34.150428745Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:35.31508464Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:35.57680288Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:35.577283581Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:36.637225412Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:36.637853481Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:36.921446108Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:37.167921175Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:37.168360578Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:39.219417347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:39.219784659Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:40.692431741Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:40.692744984Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:31:40.870809802Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:41.221326929Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:41.221773093Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:41.488797091Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:41.522101286Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:41.522434137Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:41.572241521Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:41.572604498Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:41.73789565Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:42.583224144Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:42.583647894Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:43.858718778Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:43.998872751Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:44.337803022Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:45.170709045Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:45.171020135Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:45.358126414Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:45.359032958Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:45.727773722Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:45.72848687Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:46.020315974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:46.021353558Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:46.374582501Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:46.60714831Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:46.607592066Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:46.98176962Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:47.223608042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:47.223949036Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:48.257258863Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:48.555384576Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:48.642141909Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:48.647764284Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:49.20199597Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:49.556973333Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:49.557216713Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:49.641631144Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:31:50.859473138Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:51.615173458Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:51.615919275Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:51.897978289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:51.898535125Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:51.982390553Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:52.082328361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:52.082766647Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:52.57720625Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:52.692319518Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:52.998799673Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:52.999227257Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:53.324730513Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:53.897550408Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:54.971481064Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:54.971718252Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:55.296185699Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:55.296971655Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:55.413844057Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:55.413997784Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:55.75019336Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:55.751108357Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:56.405832239Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:58.287545716Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:58.85476599Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:59.294493231Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:59.294861268Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:59.72278Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:31:59.723054484Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:00.040623326Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:00.040811856Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:32:00.859433176Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:00.930796014Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:00.991003885Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:00.991522906Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:01.833936589Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:02.348656004Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:02.348841908Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:02.693034055Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:02.69375577Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:03.792574365Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:04.679338131Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:04.721454021Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:07.179328624Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:07.2095036Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:07.229462045Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:07.229754203Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:07.255328358Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:07.256093254Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:08.011508089Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:08.515817034Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:08.823493109Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:08.823973561Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:09.691324996Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:09.691748858Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:10.016254808Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:10.452570015Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:32:10.860737979Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:12.365379588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:12.36616255Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:12.516524245Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:14.844008349Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:16.843144254Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:16.843627027Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:17.077926307Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:17.418020345Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:17.418314986Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:18.208017364Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:19.405602237Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:19.700548123Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:19.701069405Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:32:20.859439811Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:21.20918777Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:21.209567244Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:21.919714054Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:21.919890465Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:22.77432727Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:23.14062874Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:23.14088753Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:25.702646091Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:25.743727676Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:25.743967816Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:26.275703704Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:26.275975556Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:27.380098131Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:27.380402054Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:28.173217881Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:28.173863676Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:28.746772467Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:28.747319102Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:28.94950679Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:32:30.859427338Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:31.618118083Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:31.61874407Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:31.786937326Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:31.78720791Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:31.956231373Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:31.956599106Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:32.364382425Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:32.364742258Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:32.604774431Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:32.605194541Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:32.905542893Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:33.214315216Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:33.214645934Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:33.795317146Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:33.796035224Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:33.819712679Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:33.820935212Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:33.823271602Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:34.150587203Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:34.150831478Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:35.310964997Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:35.577431275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:35.577896098Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:36.600450344Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:36.600844867Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:36.919868995Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:37.160706301Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:37.161150614Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:39.110426279Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:39.110815228Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:40.735479533Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:40.736100964Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:32:40.859615535Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:41.265860576Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:41.266147929Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:41.491556042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:41.52425986Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:41.53205168Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:41.572592364Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:41.57290272Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:41.718093774Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:42.481505216Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:42.481935859Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:43.863966073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:43.998147418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:44.337616782Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:45.113940105Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:45.114197893Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:45.358134397Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:45.358960812Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:45.712066608Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:45.71277303Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:45.922631144Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:45.923238482Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:46.375178427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:46.667316078Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:46.667633874Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:46.985626164Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:47.214283457Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:47.214597912Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:48.26162555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:48.521910443Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:48.522974702Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:48.554002598Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:49.200337086Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:49.555017858Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:49.55524969Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:49.634556064Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:32:50.859464789Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:51.619894367Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:51.620594344Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:51.898827066Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:51.89934009Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:51.98373187Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:52.07621585Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:52.076627203Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:52.577649817Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:52.690592243Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:52.999366821Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:52.999820118Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:53.30097141Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:53.900641411Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:54.974458658Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:54.974736788Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:55.30402562Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:55.304413237Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:55.413113258Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:55.413250259Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:55.761675043Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:55.762104411Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:56.410533106Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:58.283020252Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:58.853545606Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:59.283075856Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:59.283457016Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:59.72446812Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:59.72478715Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:59.876916677Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:32:59.877054007Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:33:00.859409091Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:00.930341196Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:01.05931001Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:01.059912455Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:01.878134758Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:02.353389315Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:02.353733139Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:02.758323271Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:02.759867902Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:03.699516462Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:04.675458701Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:04.720331529Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:07.175131585Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:07.206859344Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:07.227999336Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:07.228235568Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:07.252480154Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:07.253248396Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:08.01305581Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:08.094776421Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:08.956504521Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:08.957053126Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:09.706659204Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:09.707154445Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:10.027379299Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:10.432977309Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:33:10.859445393Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:12.356135757Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:12.35672703Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:12.597448038Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:14.847434146Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:16.604890656Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:16.605194614Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:16.84086747Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:16.841360007Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:17.170589328Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:18.165562735Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:19.408574782Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:19.685085028Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:19.68557731Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:33:20.859480614Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:21.223955855Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:21.224310111Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:21.920925414Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:21.921103287Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:22.779288618Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:23.144605148Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:23.144870502Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:25.697260404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:25.746513073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:25.746778636Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:26.271716077Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:26.272018783Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:27.383038139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:27.383333484Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:28.160074884Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:28.160658444Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:28.771756789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:28.772257519Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:28.958425497Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:33:30.859448781Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:31.616956312Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:31.617694427Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:31.786720842Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:31.786973041Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:32.28171394Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:32.282113913Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:32.346452563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:32.346814Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:32.572971289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:32.57336114Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:32.905580923Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:33.21547289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:33.215780878Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:33.795296209Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:33.796054222Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:33.815697305Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:33.816834376Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:33.819040241Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:34.151654662Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:34.151971432Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:35.309385065Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:35.576692152Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:35.577119734Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:36.662222982Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:36.662808011Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:36.92736541Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:37.160768428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:37.16121457Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:39.189687697Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:39.190168053Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:40.802913188Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:40.803266777Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:33:40.859955881Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:41.25347687Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:41.253793964Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:41.492552384Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:41.506142451Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:41.506526865Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:41.558322659Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:41.558617206Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:41.716561856Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:42.548160401Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:42.548598995Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:43.861367357Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:43.997994352Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:44.337717959Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:44.747807991Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:44.748091721Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:45.358152108Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:45.358869663Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:45.725140419Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:45.725833135Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:45.915645148Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:45.916288249Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:46.374512545Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:46.693037748Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:46.693574484Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:46.983091805Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:47.075871568Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:47.076232393Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:48.255807036Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:48.516216229Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:48.517035682Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:48.560299243Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:49.200618912Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:49.557065782Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:49.557288783Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:49.648312647Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:33:50.859480262Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:51.632440255Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:51.633098948Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:51.903490959Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:51.904080505Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:51.982523193Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:52.082161251Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:52.082553723Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:52.577357262Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:52.690921654Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:53.002294743Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:53.002771951Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:53.301050622Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:53.896370233Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:54.980821046Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:54.981025209Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:55.305440448Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:55.305886896Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:55.412890883Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:55.413040347Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:55.650996286Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:55.651485161Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:56.402620666Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:58.295605319Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:58.856496706Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:59.277000062Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:59.277313674Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:59.729301025Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:59.72956463Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:59.870918551Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:33:59.871088644Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:34:00.859442394Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:00.931053295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:01.095826889Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:01.09648197Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:01.84144363Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:02.367176739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:02.367320052Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:02.675965494Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:02.676890708Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:03.70034341Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:04.726923488Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:04.777174147Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:07.17092176Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:07.200879342Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:07.221020866Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:07.221278893Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:07.252508539Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:07.253297998Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:08.017955422Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:08.500894448Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:09.474990599Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:09.475446228Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:09.714366184Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:09.714981606Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:10.016838469Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:10.431546854Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:34:10.859481552Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:12.384412743Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:12.384968616Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:12.521251728Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:14.867885258Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:16.724160956Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:16.724599286Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:17.044567897Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:17.411309886Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:17.411591329Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:18.174207688Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:19.40648412Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:19.689227204Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:19.689745469Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:34:20.859434486Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:21.207840992Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:21.208219347Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:21.905034606Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:21.905178005Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:22.757442331Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:23.13500743Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:23.135266597Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:25.698101093Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:25.726197503Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:25.726438751Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:26.293396422Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:26.293771902Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:27.380135662Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:27.380418963Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:28.168251574Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:28.168968388Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:28.76712898Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:28.767651927Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:28.939913522Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:34:30.859433914Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:31.621458556Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:31.622000072Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:31.786869563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:31.787109267Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:32.27983206Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:32.280255159Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:32.453497187Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:32.453850049Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:32.592855127Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:32.59330433Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:32.905718672Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:33.199484267Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:33.199908589Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:33.79526944Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:33.796006059Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:33.816125Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:33.817296672Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:33.819484483Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:34.150595272Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:34.15085057Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:35.31084438Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:35.590345481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:35.590892146Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:36.621346389Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:36.622012848Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:36.920290454Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:37.170480383Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:37.170980005Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:39.169180493Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:39.169503993Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:40.690468793Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:40.690878478Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:34:40.86046929Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:41.266234628Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:41.266602883Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:41.436221968Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:41.436690695Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:41.486540134Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:41.565427056Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:41.565794598Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:41.719138775Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:42.58626275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:42.586727475Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:43.864523107Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:43.998213791Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:44.272647509Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:45.173620358Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:45.173916404Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:45.358026839Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:45.358737429Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:45.713507946Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:45.714209698Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:45.84059945Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:45.841231142Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:46.374214605Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:46.598681585Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:46.599060375Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:46.985985788Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:47.061973662Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:47.062280662Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:48.256634987Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:48.523850246Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:48.524390138Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:48.555923479Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:49.20127693Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:49.49320579Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:49.493439473Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:49.650840158Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:34:50.859420957Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:51.626400105Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:51.627124245Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:51.897944936Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:51.898441844Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:51.984168376Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:52.068177488Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:52.068594788Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:52.577832064Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:52.691284494Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:52.998978336Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:52.999413676Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:53.310717664Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:53.896752186Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:54.975998427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:54.976210659Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:55.314223011Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:55.31468345Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:55.430526605Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:55.430709647Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:55.654686355Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:55.65512163Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:56.403165296Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:58.295810912Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:58.851853999Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:59.286158546Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:59.286487153Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:59.726585355Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:34:59.726867373Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:00.041500169Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:00.041676961Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:35:00.859418498Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:00.930933621Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:01.097982418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:01.098473543Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:02.571615263Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:02.571816167Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:03.009787597Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:03.010878236Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:03.028245237Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:03.699818113Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:04.727502234Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:04.774905599Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:07.184358021Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:07.215329623Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:07.24700524Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:07.247294302Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:07.267652337Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:07.268358463Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:08.027909768Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:08.27182598Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:09.703174321Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:09.704578949Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:09.723095819Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:09.724104713Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:10.018963638Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:10.433872389Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:35:10.859452258Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:12.352492898Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:12.353117154Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:12.516273171Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:14.842443022Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:16.723235325Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:16.724235121Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:17.077943291Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:17.471991833Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:17.472584761Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:18.145845785Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:19.408254796Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:19.688962321Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:19.689474041Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:35:20.859420322Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:21.207378126Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:21.207775626Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:21.913926996Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:21.914092018Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:22.777079125Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:23.13531833Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:23.135588198Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:25.697958315Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:25.7266461Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:25.726866129Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:26.292752011Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:26.293023587Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:27.380941011Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:27.381229873Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:28.167298527Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:28.167973604Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:28.789573575Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:28.790290222Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:28.93422978Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:35:30.859432534Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:31.616746413Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:31.617300275Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:31.786822492Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:31.787031721Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:32.296181428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:32.29658248Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:32.490076385Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:32.490515714Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:32.592032474Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:32.59248502Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:32.905558103Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:33.196558367Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:33.196881316Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:33.795359977Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:33.7960922Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:33.815938414Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:33.817079419Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:33.8192298Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:34.161429399Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:34.16171642Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:35.309168843Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:35.576637438Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:35.57726085Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:36.645497833Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:36.646050152Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:36.921279789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:37.160726261Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:37.161197414Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:39.112923346Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:39.11339827Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:40.698442301Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:40.699073442Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:35:40.860262098Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:41.378598562Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:41.379084183Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:41.486603931Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:41.528136023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:41.528462498Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:41.568944847Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:41.569252435Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:41.716942713Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:42.211399379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:42.211856268Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:43.857424213Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:44.001074421Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:44.337793414Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:44.549760939Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:44.550061045Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:45.358118693Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:45.358898304Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:45.713759266Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:45.714458333Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:45.842220477Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:45.842876701Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:46.375797437Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:46.764816439Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:46.765250862Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:46.984611655Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:47.064742243Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:47.065035733Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:48.255249424Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:48.553731458Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:48.6473825Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:48.648160498Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:49.200911831Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:49.503627162Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:49.503879891Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:49.644026931Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:35:50.859436922Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:51.627091643Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:51.627803897Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:51.910442983Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:51.911210882Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:51.983398405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:52.07358615Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:52.074034823Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:52.577029377Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:52.694834434Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:52.998815886Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:52.999231147Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:53.301347158Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:53.896730326Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:54.982024946Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:54.982267033Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:55.332642353Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:55.333079352Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:55.429475349Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:55.429653006Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:55.749290684Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:55.74972783Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:56.402202975Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:58.290920183Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:58.851647279Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:59.293058852Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:59.293383859Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:59.723462949Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:59.723779392Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:59.889513963Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:35:59.889677829Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:36:00.859428943Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:00.930941857Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:01.054348312Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:01.05521701Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:01.649409192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:02.397524085Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:02.397749052Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:02.6750833Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:02.67576196Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:03.702109034Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:04.677642131Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:04.722312407Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:07.178872611Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:07.210073227Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:07.231357304Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:07.23157109Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:07.256597807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:07.25738206Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:08.012139575Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:09.300990593Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:09.542311184Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:09.54290724Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:09.684266704Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:09.685219326Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:10.058435151Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:10.431824287Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:36:10.859447275Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:12.341471106Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:12.342084046Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:12.596829042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:14.845297003Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:16.841653661Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:16.842136093Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:17.074046594Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:17.470089734Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:17.47042296Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:18.153478815Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:19.409638594Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:19.691519345Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:19.692287971Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:36:20.859426138Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:21.208988837Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:21.209540551Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:21.921802314Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:21.921955356Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:22.773175492Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:23.136255785Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:23.136507541Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:25.702580086Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:25.726288029Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:25.726521433Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:26.272284224Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:26.272559533Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:27.379083067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:27.379354171Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:28.159250006Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:28.164334275Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:28.576918516Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:28.577462632Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:28.986632068Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:36:30.859446118Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:31.637406569Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:31.638046379Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:31.797829116Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:31.798154534Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:32.067875772Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:32.068336419Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:32.443901059Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:32.444283927Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:32.592720063Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:32.593178026Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:32.90548471Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:33.19623155Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:33.196520058Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:33.795241766Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:33.795960454Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:33.816027717Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:33.817149514Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:33.819287474Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:34.155825698Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:34.156086212Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:35.310235127Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:35.586339251Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:35.586804022Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:36.60853577Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:36.608938628Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:36.920852033Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:37.167940998Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:37.168382801Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:39.244475258Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:39.244908645Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:40.70120157Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:40.701642389Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:36:40.859518066Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:41.24464817Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:41.244965972Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:41.488465977Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:41.506778761Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:41.507115019Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:41.555788167Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:41.556211017Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:41.721154539Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:42.549072023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:42.549555806Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:43.860672169Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:44.000630649Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:44.339240039Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:45.119346511Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:45.119694962Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:45.35817732Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:45.358913413Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:45.714275332Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:45.714939208Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:45.845303745Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:45.845929389Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:46.375730414Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:46.616519052Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:46.616892279Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:46.985575177Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:47.069920326Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:47.070326887Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:48.258525444Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:48.554492622Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:48.649201229Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:48.649968145Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:49.201784296Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:49.556905718Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:49.557126618Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:49.732896813Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:36:50.859504046Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:51.625338728Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:51.626035871Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:51.92696418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:51.927468385Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:51.984126435Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:52.07708582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:52.077524815Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:52.578632373Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:52.691511986Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:52.998651464Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:52.99904676Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:53.314221016Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:53.898844585Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:54.980393614Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:54.980609098Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:55.335646168Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:55.336099257Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:55.418297787Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:55.418483218Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:55.747376275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:55.74782284Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:56.404138165Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:58.286729054Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:58.854196975Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:59.127038517Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:59.127404816Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:59.723078268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:59.723347014Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:59.897632713Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:36:59.897779438Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:37:00.859435635Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:00.930285502Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:01.03426963Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:01.034842411Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:01.554433712Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:02.378051819Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:02.378246358Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:02.696181226Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:02.696897964Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:03.698698327Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:04.673924736Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:04.720062233Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:07.291231448Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:07.293134685Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:07.348280055Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:07.45599195Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:07.491341154Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:07.491916896Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:08.03050865Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:09.443977927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:09.464429508Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:09.465923485Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:09.860437662Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:09.869895525Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:10.092230171Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:10.431611244Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:37:10.85942836Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:12.340538167Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:12.341156635Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:12.519945192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:14.840898733Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:16.84614545Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:16.846575444Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:17.042016469Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:17.413835151Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:17.414153559Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:18.169336276Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:19.435139691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:19.687271871Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:19.687816122Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:37:20.859428093Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:21.208207376Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:21.208614803Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:21.931100057Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:21.931737292Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:22.758808916Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:23.13523535Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:23.135481878Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:25.699005485Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:25.727402866Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:25.727620028Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:26.272040943Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:26.272312403Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:27.332364974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:27.332628474Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:28.199216404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:28.200089824Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:28.6411384Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:28.641793855Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:28.936193235Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:37:30.859426377Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:31.786722216Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:31.786957362Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:31.96508299Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:31.965493682Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:32.468160258Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:32.468561816Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:32.507755103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:32.508312142Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:32.592524904Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:32.593011925Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:32.905459602Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:33.206428943Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:33.206819554Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:33.795213463Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:33.795898198Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:33.815325044Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:33.816435813Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:33.818624688Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:34.15033415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:34.150597916Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:35.31072496Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:35.576839614Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:35.577312841Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:36.62765245Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:36.628155234Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:36.921201802Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:37.160880217Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:37.161357138Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:39.179155151Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:39.179521748Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:40.685835645Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:40.686084094Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:37:40.859510326Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:41.246857789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:41.247159892Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:41.427593848Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:41.427972968Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:41.487136988Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:41.556583864Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:41.556989894Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:41.71428458Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:42.54233849Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:42.54276978Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:43.859155388Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:43.997881829Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:44.341188288Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:44.549634469Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:44.54997326Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:45.358104733Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:45.358827705Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:45.713811936Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:45.714472174Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:45.852510739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:45.853126027Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:46.375026042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:46.60354105Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:46.603904974Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:46.982284275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:47.064339746Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:47.064717973Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:48.256596876Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:48.557011773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:48.64812698Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:48.648882732Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:49.205572921Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:49.557471875Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:49.557776251Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:49.646262512Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:37:50.859432054Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:51.626204377Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:51.626923081Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:51.900364268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:51.900909816Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:51.982144972Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:52.095547655Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:52.096014423Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:52.57726628Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:52.696335967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:52.998862657Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:52.999302291Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:53.300772812Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:53.899232022Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:54.973114284Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:54.973307142Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:55.339738491Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:55.340191579Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:55.412166799Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:55.41231205Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:55.756394646Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:55.756982345Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:56.402654526Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:58.269208792Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:58.858396545Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:59.097819729Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:59.098143707Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:59.742468731Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:59.742866303Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:59.88671058Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:37:59.88685995Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:38:00.859437899Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:00.930282533Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:00.997397708Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:00.997845933Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:01.743247997Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:02.367876205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:02.368060495Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:02.699534988Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:02.700234993Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:03.698762184Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:04.673681872Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:04.720323592Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:07.184478287Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:07.216168969Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:07.240056884Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:07.240350853Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:07.258251573Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:07.259203381Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:08.01708217Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:08.246832295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:09.421507807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:09.422255818Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:09.696060739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:09.69683616Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:10.061108851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:10.431902088Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:38:10.859452312Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:12.354147875Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:12.3547366Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:12.595938093Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:14.84088944Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:16.717040196Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:16.717476754Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:17.079918952Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:17.459248688Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:17.459572076Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:18.343866227Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:19.40736893Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:19.998612106Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:19.999114546Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:38:20.859451477Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:21.206481704Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:21.206933158Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:21.907628527Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:21.907832171Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:22.773219746Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:23.134856244Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:23.135133547Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:25.69748041Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:25.726505051Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:25.726752402Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:26.275650461Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:26.275941757Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:27.378681416Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:27.3789851Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:28.187342269Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:28.189234746Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:28.550083379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:28.55060669Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:28.940593356Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:38:30.859456537Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:31.786940052Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:31.787187928Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:31.962930643Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:31.963309336Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:32.3511887Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:32.351588383Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:32.446346947Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:32.44687403Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:32.59481312Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:32.595298019Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:32.905534824Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:33.196363598Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:33.19667516Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:33.795216518Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:33.795986851Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:33.816010522Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:33.817162224Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:33.819338909Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:34.150458333Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:34.150749098Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:35.310061427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:35.577182676Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:35.577680962Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:36.624089747Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:36.624571565Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:36.924770644Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:37.160753914Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:37.161268914Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:39.397858077Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:39.398358192Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:40.685865495Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:40.686140209Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:38:40.859468513Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:41.256390146Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:41.257073207Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:41.491461864Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:41.530955955Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:41.531321767Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:41.558066518Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:41.558344Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:41.710774583Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:42.548333454Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:42.54880993Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:43.866081786Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:44.002461555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:44.272379602Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:45.140794448Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:45.141094687Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:45.358088746Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:45.358823309Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:45.828197064Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:45.828897963Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:45.861471857Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:45.862018074Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:46.375215837Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:46.591160629Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:46.591509931Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:46.986119385Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:47.073136057Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:47.073470923Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:48.256723051Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:48.557737788Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:48.659316372Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:48.660132008Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:49.237607117Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:49.556682835Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:49.556890831Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:49.640389418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:38:50.859446776Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:51.615788227Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:51.616519147Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:51.901391426Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:51.902018198Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:51.985113875Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:52.066370101Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:52.06677312Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:52.575965827Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:52.711337786Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:52.998875802Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:52.999268281Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:53.300859753Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:53.90147071Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:54.982826763Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:54.983045278Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:55.305307197Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:55.305809906Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:55.413223509Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:55.413372491Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:55.765998781Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:55.766399197Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:56.411518879Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:58.258184534Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:58.854243537Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:59.280182102Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:59.280531026Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:59.723846643Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:59.724094503Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:59.887176563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:38:59.887326Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:39:00.859436337Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:00.931520114Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:01.037363573Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:01.037869304Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:01.564168957Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:02.345339817Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:02.34550143Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:02.783028912Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:02.783765319Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:03.69780231Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:04.681592619Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:04.720458643Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:07.176263472Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:07.210877342Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:07.232295394Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:07.232504762Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:07.287220249Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:07.288059201Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:08.011888628Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:08.231715238Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:09.435080573Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:09.43535321Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:09.747997406Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:09.749195631Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:10.055309858Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:10.435473468Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:39:10.862847795Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:12.353946136Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:12.354527872Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:12.596644049Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:14.84551945Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:16.714582775Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:16.715046642Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:17.073023014Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:17.443043849Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:17.443363174Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:18.249938339Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:19.408426759Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:19.680167719Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:19.680688331Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:39:20.859423227Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:21.207256305Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:21.207621051Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:21.948393983Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:21.948583261Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:22.774072881Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:23.135569924Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:23.13587642Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:25.698054609Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:25.726568775Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:25.726808405Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:26.271731124Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:26.27204217Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:27.380650117Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:27.380913323Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:28.173079192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:28.173571673Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:28.686201187Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:28.68663012Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:28.932573313Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:39:30.859426787Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:31.641140358Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:31.641739145Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:31.786731825Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:31.786951477Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:32.311010392Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:32.311427537Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:32.355528436Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:32.355910115Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:32.592935915Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:32.593388679Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:32.905742829Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:33.197323333Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:33.197604179Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:33.795210805Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:33.795915822Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:33.815647372Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:33.816761588Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:33.81891021Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:34.151625154Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:34.15191313Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:35.308992532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:35.57698795Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:35.577421632Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:36.61654322Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:36.61710187Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:36.925427812Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:37.175980387Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:37.176375664Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:39.084151716Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:39.084486692Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:40.689058062Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:40.689344166Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:39:40.859455027Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:41.243487154Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:41.243818599Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:41.502146098Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:41.557368056Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:41.558000668Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:41.589069697Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:41.589409435Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:41.740595519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:42.604909395Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:42.605442087Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:43.8589895Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:43.999236192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:44.272246099Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:45.130778175Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:45.131192288Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:45.358107826Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:45.358960507Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:45.713771824Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:45.714429702Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:45.839098349Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:45.839706076Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:46.374234921Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:46.606803668Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:46.607154056Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:46.985791994Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:47.085115221Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:47.085443864Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:48.256572198Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:48.555432423Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:48.73496224Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:48.73557057Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:49.215397041Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:49.556968479Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:49.557210871Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:49.659926826Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:39:50.859444305Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:51.649300987Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:51.649966778Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:51.893498694Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:51.89402315Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:51.979521369Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:52.072720623Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:52.073136086Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:52.576809325Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:52.691728599Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:52.998805931Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:52.999251228Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:53.342743192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:53.896732386Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:54.988108656Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:54.988318705Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:55.296025327Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:55.296473396Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:55.411941717Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:55.412100886Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:55.764719355Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:55.765119461Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:56.403094461Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:58.263459299Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:58.851483908Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:59.281027049Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:59.281352174Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:59.728101265Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:59.728405861Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:59.885671739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:39:59.885823746Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:40:00.85942895Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:00.931511673Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:01.016938416Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:01.017396765Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:01.546111925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:02.53044334Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:02.530638699Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:03.877718209Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:04.034478225Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:04.270001376Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:04.72706106Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:04.783886228Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:07.180008485Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:07.221061922Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:07.245328006Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:07.245642987Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:07.267773723Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:07.268575612Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:08.012832703Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:08.927874119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:09.082569368Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:09.08316434Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:09.711774203Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:09.712443229Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:10.028950494Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:10.461974158Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:40:10.859437648Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:12.341653773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:12.342305408Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:12.51972755Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:14.842948671Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:17.072175961Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:17.135617426Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:17.136236199Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:17.584240403Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:17.584635985Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:18.170025729Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:19.417322159Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:20.00222637Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:20.002731877Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:40:20.859410411Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:21.647935379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:21.648295473Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:21.912328109Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:21.912522474Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:22.767526294Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:23.134560447Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:23.134848414Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:25.698355202Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:25.730538948Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:25.730794352Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:26.342210466Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:26.342537686Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:27.327453755Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:27.327776231Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:28.169107174Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:28.169947336Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:28.699619907Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:28.70015498Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:28.95429449Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:40:30.859434555Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:31.636710302Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:31.637362153Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:31.78763199Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:31.787965778Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:32.285675241Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:32.286113993Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:32.351144481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:32.351570466Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:32.593222743Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:32.59381107Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:32.905560338Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:33.196434371Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:33.196764608Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:33.795235493Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:33.796032932Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:33.817331746Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:33.818517357Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:33.820752333Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:34.150468622Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:34.150710154Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:35.310161017Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:35.576850583Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:35.577328923Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:36.624776228Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:36.625254627Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:36.921529703Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:37.164937016Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:37.165384308Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:39.533928442Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:39.53434169Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:40.701185031Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:40.701447542Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:40:40.859430553Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:41.260104502Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:41.260388371Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:41.487484763Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:41.517757557Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:41.518032566Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:41.558800177Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:41.55912629Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:41.735130963Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:42.558844296Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:42.559277586Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:43.878256511Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:44.0001989Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:44.272890933Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:44.552275115Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:44.552609861Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:45.358095031Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:45.358914969Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:45.71616035Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:45.716861774Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:45.845287345Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:45.845948196Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:46.387702278Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:46.617156975Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:46.617522869Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:46.983967409Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:47.078749132Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:47.079138384Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:48.350318642Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:48.578084162Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:48.647790329Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:48.648273359Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:49.209134866Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:49.497483944Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:49.497869851Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:49.643942358Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:40:50.859486909Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:51.653634037Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:51.654372093Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:51.905157108Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:51.905741039Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:51.985311044Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:52.070793696Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:52.071209233Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:52.580533868Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:52.697084406Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:53.002694762Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:53.003159074Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:53.32602314Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:53.905590519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:54.973546904Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:54.97383664Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:55.309491388Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:55.309942114Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:55.413056558Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:55.413179791Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:55.644940666Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:55.645368537Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:56.402204862Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:58.275278446Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:58.856262121Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:59.116484502Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:59.116867133Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:59.723140265Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:59.723403827Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:59.870913926Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:40:59.871051276Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:41:00.8594368Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:00.932244249Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:01.002785527Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:01.003205629Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:01.673559395Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:02.425478141Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:02.425637401Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:02.674238328Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:02.674949057Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:03.698939059Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:04.683724068Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:04.723214156Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:07.17692803Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:07.207730288Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:07.226617344Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:07.226858408Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:07.252933838Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:07.25365784Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:08.011254834Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:08.545445596Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:08.845182351Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:08.845587118Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:09.741463217Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:09.742620943Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:10.044145892Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:10.431915167Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:41:10.859538976Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:12.415219561Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:12.416638665Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:12.52078297Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:14.846271733Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:16.842737236Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:16.843518478Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:17.171509037Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:17.455762933Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:17.456146768Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:18.263612146Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:19.409656789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:19.7032441Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:19.703738094Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:41:20.859467761Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:21.223524289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:21.223910721Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:21.915336356Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:21.915516398Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:22.758784557Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:23.134903687Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:23.135155014Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:25.698303713Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:25.726709042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:25.726952745Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:26.271950428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:26.272208221Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:27.378639389Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:27.379394152Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:28.160066052Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:28.160680411Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:28.837551285Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:28.837990912Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:28.93301145Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:41:30.859423882Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:31.643885204Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:31.644442339Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:31.786959296Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:31.787226263Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:32.275652367Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:32.276062286Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:32.348812764Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:32.349343446Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:32.592433819Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:32.592906146Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:32.905533622Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:33.19664426Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:33.196971028Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:33.795374908Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:33.796139385Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:33.816526776Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:33.817710785Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:33.819925145Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:34.150149462Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:34.150374386Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:35.310385963Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:35.577171701Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:35.577860606Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:36.704237634Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:36.704901434Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:36.939500893Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:37.170746153Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:37.171331887Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:39.296362172Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:39.297638075Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:40.685734414Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:40.686021124Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:41:40.859441771Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:41.247991569Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:41.24831576Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:41.487992992Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:41.518268626Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:41.518636434Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:41.568622142Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:41.568970171Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:41.722281692Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:42.542007722Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:42.542464109Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:43.857765283Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:43.999096925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:44.33821501Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:45.133195348Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:45.133480481Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:45.358127957Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:45.359083395Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:45.725521103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:45.726180787Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:45.841256043Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:45.841885768Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:46.375412148Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:46.63670625Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:46.637106967Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:46.986501502Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:47.072692624Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:47.073005155Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:48.265256957Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:48.562193899Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:48.644117243Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:48.644966023Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:49.20180771Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:49.503766068Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:49.504045124Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:49.738599718Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:41:50.859436669Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:51.617932961Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:51.61863376Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:51.904378186Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:51.904930293Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:51.983787152Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:52.073769726Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:52.074172197Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:52.577751072Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:52.690610157Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:52.99896633Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:52.999355588Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:53.328642103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:53.897094998Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:54.971069559Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:54.971302722Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:55.295887818Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:55.296336005Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:55.411997552Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:55.412147333Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:55.636143402Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:55.636566191Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:56.402839785Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:58.260293627Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:58.851466787Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:59.294489015Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:59.294842094Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:59.723155749Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:41:59.723433694Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:00.074727396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:00.074894831Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:42:00.859430616Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:00.931640882Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:00.985085601Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:00.985593302Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:01.948929135Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:02.381478548Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:02.381785869Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:02.676149044Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:02.676854726Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:03.702413243Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:04.6736255Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:04.723224338Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:07.171357498Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:07.202083247Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:07.221587257Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:07.221874582Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:07.258564241Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:07.259298018Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:08.014234568Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:08.820236652Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:09.469584446Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:09.470054291Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:09.734504519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:09.73550884Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:10.092811915Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:10.431331201Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:42:10.859436704Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:12.353216832Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:12.35381529Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:12.596615489Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:14.843059749Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:16.848024813Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:16.848454749Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:17.046434606Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:17.428545969Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:17.428988301Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:18.149599907Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:19.411787531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:19.686087839Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:19.686603006Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:42:20.859434843Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:21.208370561Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:21.208779696Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:21.919610952Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:21.919790537Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:22.773091962Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:23.134400087Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:23.134653232Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:25.698003027Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:25.750747544Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:25.750979036Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:26.273477151Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:26.273984511Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:27.386081736Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:27.386386302Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:28.180621143Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:28.181249334Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:28.79736865Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:28.797940214Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:28.955742844Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:42:30.859435186Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:31.614727749Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:31.615357967Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:31.791630178Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:31.791892502Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:32.287755422Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:32.288148026Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:32.451176447Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:32.45154175Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:32.591126879Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:32.591542364Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:32.905443284Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:33.196219935Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:33.196557167Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:33.795361378Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:33.796087858Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:33.816598544Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:33.818189032Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:33.820606299Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:34.152065005Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:34.152339913Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:35.30974616Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:35.577863851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:35.578314473Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:36.668428515Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:36.668923644Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:36.920304046Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:37.161343676Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:37.161932685Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:39.187366851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:39.187910284Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:40.701027263Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:40.701271851Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:42:40.859419236Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:41.253358647Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:41.253812673Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:41.491483585Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:41.531450215Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:41.531902905Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:41.557364603Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:41.557653033Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:41.725381376Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:42.222927453Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:42.223401193Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:43.863972387Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:43.998760653Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:44.340534808Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:45.12838088Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:45.128713517Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:45.358072028Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:45.35886724Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:45.714347008Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:45.715080632Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:45.853532845Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:45.854173201Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:46.377575489Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:46.626522307Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:46.62711419Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:46.982226164Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:47.060874606Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:47.061191328Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:48.258352078Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:48.520549836Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:48.520995325Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:48.552980855Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:49.202549802Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:49.556830102Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:49.557112672Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:49.641269527Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:42:50.859434882Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:51.614907354Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:51.615626423Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:51.894388295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:51.894979216Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:51.981204242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:52.066780833Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:52.067172527Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:52.578407803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:52.698408253Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:52.998815019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:52.999268729Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:53.328457731Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:53.897092179Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:54.975447516Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:54.975694397Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:55.332603802Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:55.333069096Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:55.412527544Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:55.412701175Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:55.756608549Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:55.757094683Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:56.402754916Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:58.269156173Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:58.85163776Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:59.273698583Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:59.274005035Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:59.72334141Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:59.723638373Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:59.933927189Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:42:59.934090277Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:43:00.859433589Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:00.930349545Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:01.009937168Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:01.010424286Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:01.786632835Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:02.375276834Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:02.375405491Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:02.714538297Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:02.715620659Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:03.69811355Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:04.682323232Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:04.720082615Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:07.182044235Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:07.212091343Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:07.231251377Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:07.231455496Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:07.270116943Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:07.271193679Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:08.009917008Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:08.331717286Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:09.47257072Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:09.477006835Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:09.703519943Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:09.707971238Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:10.060471474Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:10.432378258Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:43:10.859433145Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:12.3541337Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:12.354747427Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:12.516978066Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:14.852481741Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:16.640650347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:16.641369859Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:16.714464807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:16.714909998Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:17.076213525Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:18.192464417Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:19.405580039Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:19.693485512Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:19.694027325Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:43:20.859426773Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:21.217405317Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:21.217923695Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:21.905149052Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:21.905306402Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:22.764166335Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:23.134542065Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:23.134822408Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:25.697621032Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:25.727745508Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:25.727994085Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:26.275488799Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:26.2757928Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:27.380597708Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:27.380908625Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:28.158576184Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:28.159060798Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:28.563491679Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:28.563942321Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:28.932498666Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:43:30.859425231Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:31.612309812Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:31.613003327Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:31.786583056Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:31.786837698Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:31.963759379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:31.964188071Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:32.452010314Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:32.45236796Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:32.594010615Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:32.594436507Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:32.905721273Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:33.196627521Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:33.196983324Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:33.811814504Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:33.812567103Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:33.842147431Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:33.843521031Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:33.847005936Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:34.15076829Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:34.151032895Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:35.311675936Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:35.577022902Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:35.577477292Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:36.616632522Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:36.617070474Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:36.921002862Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:37.162877838Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:37.163336986Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:39.32883339Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:39.329325223Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:40.704427673Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:40.704738111Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:43:40.859442156Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:41.047874801Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:41.048192184Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:41.490881689Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:41.519002807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:41.519407479Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:41.554510515Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:41.554842555Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:41.735899676Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:42.539773207Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:42.540207734Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:43.85882642Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:43.997748102Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:44.338856678Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:44.982891029Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:44.983182957Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:45.358110292Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:45.358871858Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:45.730844722Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:45.731493303Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:45.844937659Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:45.845555597Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:46.375139156Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:46.609590167Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:46.609974026Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:46.984075516Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:47.211888873Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:47.212204581Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:48.256395098Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:48.511762559Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:48.51226948Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:48.553265371Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:49.200068047Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:49.554701298Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:49.554938352Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:49.66147941Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:43:50.859498291Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:51.629207175Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:51.629954079Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:51.911498053Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:51.912231102Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:51.987043278Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:52.081583092Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:52.082133913Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:52.578347684Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:52.691387228Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:52.999163582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:52.999641155Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:53.328433239Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:53.898954602Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:54.971432477Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:54.971630737Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:55.331868135Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:55.332339446Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:55.41236304Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:55.41249311Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:55.755742067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:55.756289098Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:56.402751011Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:58.289344402Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:58.904568205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:59.277904695Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:59.27822675Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:59.725833701Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:59.726225618Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:59.889147109Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:43:59.889302242Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:44:00.859424878Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:00.915241156Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:00.915784112Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:00.930908084Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:01.611234144Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:02.366640217Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:02.366817011Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:02.673157992Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:02.673840892Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:03.699316565Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:04.681151462Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:04.720809924Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:07.184532903Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:07.235301037Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:07.254587301Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:07.255410667Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:07.262615876Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:07.262945802Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:08.018881896Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:08.230918128Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:09.471203136Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:09.471616158Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:09.723708506Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:09.724440406Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:10.013424785Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:10.431288659Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:44:10.859445902Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:12.355188109Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:12.355816242Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:12.516000083Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:14.843484209Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:16.728359397Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:16.728835205Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:17.071736042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:17.41927232Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:17.420425844Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:18.188204612Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:19.410109396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:19.700405073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:19.700932567Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:44:20.859458948Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:21.217698979Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:21.218031977Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:21.916837107Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:21.917005693Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:22.773241095Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:23.176048877Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:23.176337083Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:25.697717096Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:25.728993905Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:25.729219397Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:26.294875051Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:26.29514226Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:27.311343166Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:27.311611902Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:28.163609324Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:28.165067029Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:28.566331541Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:28.566845139Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:28.964130707Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:44:30.859429873Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:31.609377104Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:31.610007584Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:31.788308773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:31.788541545Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:31.987902678Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:31.988297974Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:32.466706575Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:32.467117425Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:32.593262678Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:32.593809553Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:32.905842944Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:33.196743278Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:33.197050197Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:33.7952852Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:33.796037736Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:33.816268469Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:33.817484857Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:33.819748285Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:34.150749249Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:34.151018994Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:35.310282847Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:35.577787516Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:35.578241308Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:36.602942707Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:36.603327332Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:36.920229625Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:37.160959698Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:37.161395361Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:39.212722974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:39.213104986Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:40.686242697Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:40.68649638Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:44:40.859431565Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:41.246844025Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:41.24712031Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:41.490526652Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:41.510846707Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:41.511243548Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:41.555377916Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:41.555681306Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:41.716452898Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:42.550711664Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:42.551227688Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:43.955598671Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:44.000574547Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:44.342826174Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:45.119578264Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:45.119849109Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:45.358104333Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:45.358831899Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:45.718635209Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:45.719330117Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:45.835261187Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:45.835893263Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:46.374941759Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:46.592271039Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:46.592590791Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:46.983850193Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:47.223016658Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:47.223346895Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:48.263289795Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:48.51072619Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:48.511179315Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:48.553391332Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:49.202980555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:49.556263289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:49.556519911Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:49.653267738Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:44:50.859441498Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:51.627873811Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:51.628595076Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:51.910927858Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:51.911444156Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:51.986433309Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:52.083882241Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:52.084297905Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:52.579037966Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:52.691358932Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:52.998877263Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:52.999332464Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:53.332108246Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:53.901972969Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:55.216000304Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:55.216205409Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:55.33301945Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:55.333448421Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:55.430537529Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:55.43196987Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:55.768155515Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:55.76855006Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:56.405017519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:58.25622231Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:58.852323047Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:59.295514238Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:59.295918707Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:59.722941442Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:44:59.723189678Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:00.032114799Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:00.032275328Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:45:00.859426912Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:00.882901648Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:00.883319826Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:00.930222686Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:01.948071389Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:02.411058649Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:02.411250079Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:02.924450968Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:02.925531494Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:03.711289863Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:04.679088119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:04.725373516Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:07.183569501Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:07.21354321Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:07.23291544Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:07.233151343Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:07.266755337Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:07.267475151Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:08.018866049Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:09.227090835Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:09.22747395Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:09.788214742Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:09.788821665Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:10.029625277Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:10.414073359Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:10.43512041Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:45:10.859404322Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:12.37437257Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:12.374998703Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:12.518256096Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:14.84363005Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:16.840873029Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:16.841605548Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:17.084134492Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:17.481466401Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:17.482401174Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:18.49499902Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:19.406488851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:19.682527902Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:19.683075745Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:45:20.860205243Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:21.207835077Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:21.208168037Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:21.91339877Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:21.913580452Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:22.759525299Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:23.135051037Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:23.135310462Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:25.69755724Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:25.72639327Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:25.726680959Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:26.271860749Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:26.27216271Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:27.382555405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:27.382941167Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:28.17464082Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:28.177197184Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:28.672144661Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:28.672542332Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:28.944739135Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:45:30.85945104Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:31.645329168Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:31.645968083Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:31.786907818Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:31.787111024Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:32.276356057Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:32.276785769Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:32.457505189Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:32.457911589Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:32.592436746Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:32.592867076Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:32.905801141Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:33.196934218Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:33.197356682Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:33.795566834Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:33.796529841Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:33.828534255Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:33.830492792Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:33.833835808Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:34.150558741Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:34.150834734Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:35.309674105Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:35.603456782Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:35.603932535Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:36.616292521Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:36.616726209Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:36.947216511Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:37.160959659Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:37.161441301Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:39.32344264Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:39.323897942Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:40.700500603Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:40.700792746Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:45:40.859435798Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:41.273248449Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:41.273651358Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:41.490392972Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:41.508332295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:41.508730522Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:41.554439362Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:41.554775655Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:41.718422169Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:42.548611283Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:42.549051025Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:43.862133537Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:43.997739095Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:44.33821409Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:44.558986498Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:44.559273483Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:45.358103288Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:45.358852921Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:45.713853062Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:45.714460951Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:45.837148507Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:45.837809609Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:46.377979023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:46.763618343Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:46.764095131Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:46.983116289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:47.220231102Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:47.220504109Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:48.26051654Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:48.554570317Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:48.691277155Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:48.691917412Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:49.201299842Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:49.558009639Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:49.558245105Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:49.642945437Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:45:50.859440285Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:51.620469365Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:51.621215155Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:51.922088995Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:51.922801431Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:51.983211896Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:52.068925137Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:52.06933485Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:52.578397236Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:52.692603407Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:52.999008571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:52.999425822Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:53.346190395Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:53.896309249Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:55.00050945Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:55.000965103Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:55.351723379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:55.352321194Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:55.412485981Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:55.412635738Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:55.76937731Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:55.769831439Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:56.402507216Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:58.253616157Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:58.910761151Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:59.3024335Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:59.302835904Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:59.727247841Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:59.727483753Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:59.890101347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:45:59.890261294Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:46:00.859436153Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:00.931234195Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:01.18222686Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:01.18274332Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:02.113276184Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:02.34588554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:02.346045647Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:02.680772235Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:02.681430014Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:03.700154281Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:04.678864807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:04.721476226Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:07.17832209Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:07.209911548Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:07.230193005Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:07.230471671Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:07.242261985Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:07.243018366Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:08.01658682Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:09.461249298Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:09.461584215Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:09.6924877Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:09.693250577Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:10.023160402Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:10.208616452Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:10.435041889Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:46:10.85943741Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:12.357997288Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:12.358560066Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:12.517654384Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:14.847522126Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:16.863964792Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:16.864395399Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:17.077123969Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:17.458199456Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:17.458722716Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:18.234905511Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:19.413060023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:19.679568809Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:19.680109449Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:46:20.859458132Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:21.234555519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:21.23493114Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:21.927242618Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:21.927406549Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:22.777021669Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:23.134554506Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:23.134831331Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:25.722859569Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:25.726474551Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:25.726749661Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:26.296055505Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:26.296359205Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:27.378749647Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:27.379056202Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:28.169096546Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:28.169920236Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:28.806610759Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:28.807158708Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:28.937124067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:46:30.859430078Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:31.626843962Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:31.627381331Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:31.787365475Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:31.787593882Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:32.30184276Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:32.302230614Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:32.354209586Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:32.354586751Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:32.602211374Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:32.602698516Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:32.905578877Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:33.196304532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:33.196619542Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:33.795594866Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:33.796353817Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:33.818340278Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:33.819487285Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:33.821685531Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:34.161546803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:34.161817204Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:35.309109393Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:35.57736634Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:35.577849481Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:36.606932769Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:36.607308321Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:36.974880287Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:37.167522264Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:37.167983881Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:39.281284362Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:39.282119501Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:40.685894194Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:40.686136266Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:46:40.859514264Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:41.200363449Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:41.20080649Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:41.4880276Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:41.519804655Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:41.520136051Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:41.557572382Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:41.557920569Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:41.728418343Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:42.587699221Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:42.588262985Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:43.86257411Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:43.997822453Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:44.350162914Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:45.118906705Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:45.119191239Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:45.358156867Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:45.358926076Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:45.716622952Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:45.717283278Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:45.833576774Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:45.834170542Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:46.376119853Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:46.601933657Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:46.602268658Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:46.98418953Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:47.083385825Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:47.083719457Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:48.350978516Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:48.553806566Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:48.697521057Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:48.698225577Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:49.200723465Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:49.504520766Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:49.504806578Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:49.642018617Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:46:50.859443388Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:51.627094257Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:51.627788133Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:51.892236521Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:51.892882024Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:51.984191001Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:52.065098958Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:52.065503906Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:52.578063693Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:52.692541456Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:52.998715314Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:52.999157379Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:53.302192468Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:53.896744534Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:55.01879581Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:55.019027183Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:55.295817599Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:55.29626539Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:55.412329206Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:55.412496958Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:55.748265311Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:55.748641605Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:56.402440457Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:58.308832432Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:58.851412258Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:59.105506779Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:59.106085345Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:59.730950444Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:59.731242584Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:59.88646363Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:46:59.886611739Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:47:00.859426323Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:00.930343506Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:00.994618269Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:00.995130447Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:01.77965713Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:02.345450298Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:02.345603647Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:02.706474724Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:02.707593819Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:03.700445244Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:04.680287722Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:04.720445161Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:07.174897256Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:07.213448625Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:07.241307077Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:07.241575587Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:07.277958551Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:07.278717659Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:08.018041246Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:08.03654932Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:09.388220504Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:09.388510727Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:09.673134898Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:09.67356856Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:10.015959397Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:10.43458473Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:47:10.90528057Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:12.354549983Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:12.355160742Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:12.597214122Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:14.844627893Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:16.728382762Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:16.729288695Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:17.081154919Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:17.435837312Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:17.436609809Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:18.174251272Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:19.405691625Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:19.999430856Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:19.999957813Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:47:20.859516426Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:21.216758698Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:21.217212251Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:21.909134548Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:21.909284969Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:22.773313389Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:23.134776662Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:23.13503Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:25.69772959Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:25.72696172Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:25.727200561Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:26.271559013Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:26.27184142Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:27.311557033Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:27.311912178Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:28.165225316Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:28.165932668Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:28.730215234Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:28.730570384Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:28.946175498Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:47:30.85944478Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:31.646740812Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:31.647288359Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:31.78731861Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:31.787591696Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:32.278301668Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:32.278728279Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:32.349270119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:32.349629862Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:32.597939639Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:32.598395081Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:32.905543946Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:33.196240887Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:33.196512448Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:33.795243204Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:33.795945944Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:33.816676474Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:33.81784299Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:33.819984723Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:34.150287064Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:34.150525934Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:35.31201227Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:35.57714401Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:35.577601433Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:36.657186506Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:36.657651567Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:36.92710986Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:37.160788266Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:37.161237696Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:39.191834096Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:39.192183831Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:40.688889704Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:40.68921141Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:47:40.859509785Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:41.271052759Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:41.27145528Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:41.422273892Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:41.42265534Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:41.486321731Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:41.567191885Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:41.567654031Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:41.742481032Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:42.550706109Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:42.551166105Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:43.859965827Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:43.997184669Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:44.339195932Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:44.56371225Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:44.563978922Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:45.358261406Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:45.358970666Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:45.712898936Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:45.71354158Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:45.833929403Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:45.834505422Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:46.3757827Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:46.605837165Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:46.606201862Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:46.986897145Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:47.069529821Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:47.069885912Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:48.259385361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:48.552423433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:48.653774814Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:48.655172043Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:49.202175582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:49.503544042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:49.503816315Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:49.637377124Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:47:50.85952582Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:51.627971651Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:51.628710003Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:51.912889424Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:51.913520636Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:51.984651629Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:52.068320231Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:52.06868594Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:52.576252144Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:52.695053822Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:52.999063813Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:52.99951108Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:53.300343258Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:53.897089578Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:54.974423569Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:54.97469499Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:55.305263298Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:55.305796421Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:55.426777354Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:55.426945338Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:55.770258282Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:55.770713722Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:56.402494177Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:58.288330148Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:58.853326023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:59.112243783Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:59.112635023Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:59.730879519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:59.731112056Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:59.897716152Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:47:59.897853279Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:48:00.859425951Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:00.930394369Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:01.072259239Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:01.072793453Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:01.681832599Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:02.345553437Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:02.345763792Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:02.741211858Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:02.742187688Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:03.698087752Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:04.678611002Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:04.752340184Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:07.182455815Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:07.213457953Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:07.234118598Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:07.234443231Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:07.258818083Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:07.259557714Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:08.026248396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:08.165353889Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:09.390893269Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:09.391203982Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:09.691292959Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:09.691833118Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:10.019709546Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:10.504300892Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:48:10.861199607Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:12.34120618Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:12.341835968Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:12.596774205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:14.841151215Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:16.71478578Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:16.715246011Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:17.042433637Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:17.459594933Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:17.460004938Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:18.145679384Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:19.406649653Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:19.680649063Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:19.681332024Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:48:20.859462221Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:21.209475867Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:21.209813324Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:21.901767475Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:21.901894423Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:22.758949342Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:23.13500714Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:23.135270559Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:25.698153505Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:25.727047451Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:25.727277734Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:26.272261464Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:26.272526922Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:28.16011862Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:28.160945549Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:28.745895986Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:28.746312049Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:28.937260487Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:48:30.859445045Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:31.787412295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:31.7876954Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:32.282804883Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:32.283268327Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:32.373919268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:32.374319166Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:32.465439509Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:32.465914408Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:32.592545012Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:32.593032246Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:32.905783737Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:33.209308132Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:33.209683279Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:33.795283063Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:33.796075312Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:33.816683287Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:33.817976003Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:33.820441509Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:34.150440551Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:34.150729964Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:35.309413185Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:35.577078062Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:35.57751955Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:36.598916766Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:36.59932709Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:36.920642948Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:37.160851476Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:37.161280352Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:37.303089348Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:37.303400974Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:39.286523687Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:39.287030376Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:40.685873776Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:40.686144065Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:48:40.859442189Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:41.247005398Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:41.2473144Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:41.487001599Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:41.562591073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:41.563041644Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:41.568959669Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:41.569321436Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:41.726817836Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:42.567903929Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:42.568340491Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:43.866152764Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:43.998365408Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:44.339025184Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:45.166196376Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:45.166540457Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:45.358017385Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:45.358753402Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:45.721895549Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:45.722581801Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:45.839571126Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:45.840229718Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:46.376706683Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:46.590601217Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:46.590989312Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:46.983429398Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:47.061534813Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:47.06283595Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:48.259453955Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:48.554352093Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:48.656737403Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:48.657216751Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:49.200334102Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:49.503731724Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:49.50398347Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:49.646328579Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:48:50.859442586Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:51.614766876Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:51.615723444Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:51.911859919Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:51.912484303Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:52.079788643Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:52.102485792Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:52.102926582Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:52.577020082Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:52.69706857Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:52.998988783Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:52.999417262Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:53.314293305Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:53.896714482Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:54.980291035Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:54.980551665Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:55.296234431Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:55.296987682Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:55.430679802Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:55.430845154Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:55.701383188Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:55.701949265Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:56.403276707Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:58.284171859Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:58.857217736Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:59.282291669Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:59.28262668Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:59.723320712Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:59.723617371Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:59.888416421Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:48:59.888583281Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:49:00.859434123Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:00.931752733Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:01.006180986Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:01.006647903Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:02.175264195Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:02.345009588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:02.345172429Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:02.707215203Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:02.707917046Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:03.697312122Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:04.679297231Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:04.720120241Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:07.170830143Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:07.201611385Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:07.220301716Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:07.220517679Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:07.292442645Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:07.293204582Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:08.053620347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:08.140453227Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:08.794723411Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:08.795094075Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:09.691444973Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:09.691969657Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:10.015952998Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:10.467210432Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:49:10.859448388Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:12.355289467Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:12.355907102Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:12.595736182Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:14.844128056Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:16.845753925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:16.846143233Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:17.078507628Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:17.419768886Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:17.420101726Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:18.485490153Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:19.41396539Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:20.007317341Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:20.009023891Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:49:20.859437468Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:21.209158623Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:21.209538237Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:21.927613718Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:21.927813302Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:22.773593204Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:23.134702556Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:23.134952618Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:25.697249956Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:25.726465137Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:25.726751327Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:26.308329298Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:26.30859553Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:27.380978532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:27.381212602Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:28.164832917Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:28.170571982Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:28.776116083Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:28.776565349Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:28.935099888Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:49:30.859439208Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:31.786776085Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:31.787047226Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:31.95677899Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:31.957165105Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:32.364724137Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:32.365081655Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:32.508112246Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:32.508596289Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:32.592184955Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:32.592650402Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:32.905519137Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:33.1972358Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:33.197528646Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:33.795360639Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:33.79610254Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:33.816025681Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:33.81716118Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:33.819313266Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:34.150238851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:34.15047804Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:35.322371045Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:35.576803165Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:35.577257777Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:36.637727477Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:36.638094251Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:36.92136073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:37.160742267Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:37.161207122Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:39.167474631Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:39.16799587Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:40.685605502Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:40.68590631Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:49:40.859485727Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:41.2550231Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:41.255354258Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:41.487772198Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:41.522658237Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:41.523030409Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:41.559770187Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:41.560046771Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:41.727921342Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:42.526654221Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:42.527110226Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:43.862683955Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:44.000771176Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:44.272618692Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:45.116901562Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:45.117203129Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:45.358080663Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:45.358810599Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:45.724236076Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:45.724937501Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:45.839352841Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:45.840010498Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:46.376158121Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:46.598266549Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:46.598607318Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:46.981778751Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:47.060190791Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:47.060483879Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:48.258742017Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:48.552186077Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:48.66034157Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:48.661110332Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:49.202141223Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:49.557521699Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:49.557764718Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:49.652237685Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:49:50.859440606Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:51.623739889Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:51.624473284Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:51.932606712Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:51.933176514Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:51.987120132Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:52.093250488Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:52.093791863Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:52.57656988Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:52.690858998Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:52.998655026Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:52.99908518Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:53.315479031Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:53.896735263Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:54.980790293Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:54.98101957Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:55.346568046Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:55.347026755Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:55.411745336Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:55.41187358Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:55.648418003Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:55.648847596Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:56.402429114Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:58.299889255Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:58.852414941Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:59.274120406Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:59.274411037Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:59.72334736Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:59.723634432Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:59.889219278Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:49:59.889379788Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:50:00.859428626Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:00.93090681Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:01.056781324Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:01.057256153Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:02.6602698Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:02.67728367Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:03.817897159Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:04.295375694Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:04.296484266Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:04.726961193Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:04.772990411Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:04.892681289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:07.211136498Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:07.263452661Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:07.359168571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:07.360658573Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:07.375937033Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:07.376281408Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:08.012021904Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:08.934902363Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:08.93529253Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:09.087346833Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:09.777456072Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:09.778899369Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:10.092092098Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:10.577880927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:50:10.85945741Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:12.390018379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:12.390642868Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:12.515829326Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:14.844229674Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:16.867601281Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:16.868078685Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:17.086588558Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:17.682845106Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:17.683320522Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:18.229907427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:19.40992996Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:20.00789283Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:20.008396413Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:50:20.859430705Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:21.206292803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:21.206646405Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:22.003049562Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:22.003191528Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:22.764736603Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:23.134553376Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:23.1348161Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:25.697395683Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:25.728085363Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:25.728358083Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:26.274228Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:26.27458691Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:27.328970749Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:27.330006511Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:28.159653533Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:28.160273285Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:28.747516748Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:28.747896806Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:28.957603129Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:50:30.860062695Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:31.786837059Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:31.787109686Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:31.96618334Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:31.966529783Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:32.469303063Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:32.469762709Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:32.498431713Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:32.498906581Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:32.592716395Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:32.593128684Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:32.905744207Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:33.197128611Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:33.19740313Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:33.795635212Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:33.796615725Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:33.821241961Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:33.822526153Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:33.82499134Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:34.150462613Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:34.150781805Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:35.310634627Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:35.577675037Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:35.57814899Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:36.630341511Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:36.63074452Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:36.925721163Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:37.160789305Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:37.161242787Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:39.317965311Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:39.3184308Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:40.68600688Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:40.68625888Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:50:40.859430227Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:41.27089023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:41.271489808Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:41.49003504Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:41.510245959Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:41.510579608Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:41.579202301Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:41.57962754Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:41.721060782Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:42.556997961Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:42.557401901Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:43.866023408Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:43.999956493Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:44.337848378Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:44.816732531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:44.817161552Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:45.358095Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:45.358864225Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:45.712842405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:45.713476317Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:45.837084883Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:45.837727965Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:46.376726Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:46.596452358Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:46.596991416Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:46.989060136Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:47.064293094Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:47.064849805Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:48.265422268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:48.552746754Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:48.649238285Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:48.650044814Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:49.199460301Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:49.55996367Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:49.560831814Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:49.679566958Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:50:50.859459164Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:51.656522947Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:51.657237836Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:51.89935672Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:51.900014502Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:52.07202496Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:52.072511383Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:52.082910626Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:52.577350332Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:52.691571519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:53.010579349Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:53.011035727Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:53.314633922Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:53.912018002Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:54.981141608Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:54.981366386Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:55.379713192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:55.380437811Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:55.42970651Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:55.429846574Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:55.746340685Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:55.747229615Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:56.412400581Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:58.28726756Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:58.861646437Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:59.146437353Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:59.147037613Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:59.721360926Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:59.721619759Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:59.8865229Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:50:59.88668147Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:51:00.859430066Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:00.932137204Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:01.158184734Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:01.158761064Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:01.611304679Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:02.388421806Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:02.388578096Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:02.676314391Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:02.677066279Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:03.701943849Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:04.724291837Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:04.778231489Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:07.181582531Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:07.211455059Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:07.231776936Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:07.23202653Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:07.260769612Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:07.26149572Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:08.020089486Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:09.441628228Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:09.442128821Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:09.525954806Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:09.712240325Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:09.71363876Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:10.020542366Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:10.432644131Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:51:10.859434631Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:12.357069199Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:12.357692499Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:12.597299859Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:14.85021266Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:16.724986647Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:16.725431371Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:17.076960091Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:17.426931339Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:17.427306325Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:18.155411787Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:19.411967029Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:19.695071609Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:19.69554771Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:51:20.86004207Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:21.251055791Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:21.252024569Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:21.913385142Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:21.913560788Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:22.759597379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:23.135162008Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:23.135449137Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:25.697933192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:25.726820304Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:25.727065799Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:26.273268901Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:26.273533004Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:27.377800012Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:27.378057949Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:28.171080172Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:28.171875744Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:28.852075533Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:28.852569252Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:28.939808129Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:51:30.859426544Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:31.658842806Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:31.659301801Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:31.787974546Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:31.788250574Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:32.276547122Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:32.276999011Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:32.455404941Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:32.45575513Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:32.593604446Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:32.594075671Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:32.905552136Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:33.196747953Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:33.197066065Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:33.795284754Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:33.796012813Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:33.815462059Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:33.816576031Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:33.818717594Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:34.1508952Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:34.151156853Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:35.310864682Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:35.577055054Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:35.577502068Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:36.657986134Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:36.658539253Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:36.920704949Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:37.172809702Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:37.173242807Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:39.216594643Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:39.217138783Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:40.689527927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:40.69007099Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:51:40.859432581Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:41.250460814Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:41.250779304Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:41.487191592Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:41.510536932Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:41.511035021Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:41.556449907Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:41.556735127Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:41.720572361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:42.542153367Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:42.542591923Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:43.866778861Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:43.998772872Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:44.338360864Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:45.135228586Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:45.135512138Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:45.358097821Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:45.358821848Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:45.734212472Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:45.734978893Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:45.834364099Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:45.835019745Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:46.377233534Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:46.593681598Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:46.594142795Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:46.985608728Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:47.060389719Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:47.060721494Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:48.254142219Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:48.553619462Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:48.663307757Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:48.663916307Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:49.20090124Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:49.575094058Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:49.57535023Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:49.648941901Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:51:50.859433254Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:51.62795771Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:51.628741525Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:51.920825837Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:51.921447742Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:51.983181251Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:52.07946638Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:52.079886806Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:52.577541897Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:52.692136899Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:53.002914303Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:53.003335844Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:53.300780853Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:53.897119757Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:54.974454641Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:54.974686503Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:55.332658404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:55.33309605Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:55.412457481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:55.412613623Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:55.7688565Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:55.769285334Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:56.401815476Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:58.278814704Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:58.852981225Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:59.295070141Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:59.295379968Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:59.706446098Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:59.707340202Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:59.870330693Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:51:59.870498863Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:52:00.859436813Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:00.930722376Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:01.023958803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:01.024432268Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:01.517410153Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:02.416588201Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:02.416750864Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:02.707365418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:02.708103802Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:03.698870603Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:04.677226827Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:04.720569334Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:07.177272114Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:07.209979985Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:07.230259089Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:07.230504061Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:07.2502132Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:07.250964541Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:08.017014767Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:08.254847707Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:09.405539901Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:09.40585184Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:09.691172526Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:09.691639905Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:10.041380117Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:10.459266755Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:52:10.860183196Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:12.361725601Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:12.362309161Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:12.595973038Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:14.84147765Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:16.725196953Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:16.725680535Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:17.080233351Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:17.404589032Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:17.404890015Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:18.201452635Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:19.404288429Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:19.679806126Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:19.680332037Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:52:20.859481481Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:21.206537172Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:21.206916611Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:21.908407161Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:21.908570794Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:22.773993047Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:23.136841751Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:23.137156273Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:25.698814481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:25.726528458Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:25.726772709Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:26.273952572Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:26.274253817Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:27.378484867Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:27.37880073Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:28.176021597Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:28.176621408Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:28.540568802Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:28.541121722Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:28.961620091Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:52:30.859431893Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:31.630771042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:31.631295372Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:31.786770656Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:31.787008319Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:32.271443525Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:32.271853954Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:32.457420657Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:32.45780754Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:32.592642317Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:32.593113797Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:32.905614932Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:33.196702376Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:33.197051349Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:33.795244845Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:33.796091118Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:33.817119906Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:33.818298635Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:33.820651428Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:34.151565795Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:34.151838014Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:35.311263207Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:35.577279538Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:35.577749878Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:36.609072022Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:36.609458338Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:36.934052542Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:37.161169712Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:37.161614763Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:39.215798015Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:39.22047057Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:40.700504558Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:40.700802193Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:52:40.859432028Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:41.244849889Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:41.245166181Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:41.49215474Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:41.510499386Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:41.510969984Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:41.554881602Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:41.55518506Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:41.715772221Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:42.221032139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:42.221468328Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:43.862745715Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:43.998086389Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:44.293268464Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:44.557083342Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:44.55735041Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:45.358108519Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:45.358893349Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:45.728364574Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:45.729292494Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:45.833281417Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:45.833868778Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:46.375263998Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:46.591146318Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:46.591464129Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:46.985886672Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:47.060312781Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:47.060714287Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:48.266431667Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:48.52252346Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:48.523141825Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:48.567736387Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:49.20116892Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:49.555602315Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:49.555863075Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:49.649016976Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:52:50.859434309Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:51.616069495Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:51.616737526Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:51.914899886Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:51.915544775Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:51.984426471Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:52.072391624Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:52.072809948Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:52.576819411Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:52.69665732Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:52.998946496Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:52.99936142Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:53.314341454Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:53.898232954Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:54.972865781Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:54.973082225Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:55.336377053Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:55.336848391Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:55.471112509Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:55.471262907Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:55.737592894Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:55.738056928Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:56.402618279Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:58.28358853Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:58.851475282Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:59.288553732Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:59.288853262Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:59.705535164Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:59.705839666Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:59.920233614Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:52:59.920395768Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:53:00.859428467Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:00.93094694Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:01.074466107Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:01.074975451Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:01.568031678Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:02.369215266Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:02.369391975Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:02.681994856Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:02.68359619Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:03.698997448Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:04.680160206Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:04.719934088Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:07.177349732Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:07.20880372Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:07.234731327Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:07.23496705Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:07.261786506Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:07.262589773Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:08.015237578Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:08.149322637Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:09.372217526Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:09.372504402Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:09.69112241Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:09.691566436Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:10.014810263Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:10.430742504Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:53:10.859389386Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:12.363945667Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:12.365001221Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:12.521612222Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:14.844658988Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:16.605041468Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:16.605360554Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:16.724308257Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:16.724885247Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:17.07942699Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:18.158613062Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:19.405352671Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:19.973165084Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:19.973716989Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:53:20.859492553Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:21.209153788Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:21.209587671Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:21.95759172Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:21.95897238Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:22.788039418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:23.141957245Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:23.142421732Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:25.698152951Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:25.739924916Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:25.740183716Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:26.271865019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:26.272157353Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:27.3812652Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:27.381552404Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:28.171308508Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:28.172235111Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:28.551746115Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:28.552318134Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:28.931830673Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:53:30.859426768Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:31.653014813Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:31.653606249Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:31.792239803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:31.792514102Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:32.276684154Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:32.277092452Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:32.347056577Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:32.347405482Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:32.610125567Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:32.610568318Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:32.905561268Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:33.196363006Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:33.196656927Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:33.795337519Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:33.796084571Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:33.818020626Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:33.819286356Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:33.821642367Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:34.151026839Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:34.151283858Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:35.309383806Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:35.580345094Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:35.580953753Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:36.628936116Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:36.629365822Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:36.923724789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:37.167751427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:37.168202747Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:39.218471528Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:39.219031604Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:40.68569451Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:40.685975637Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:53:40.859448312Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:41.056696058Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:41.057059901Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:41.487318824Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:41.534741211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:41.535127736Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:41.567398744Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:41.56786725Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:41.708797904Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:42.558355609Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:42.558800299Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:43.968005812Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:43.997845876Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:44.339593245Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:45.123086725Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:45.123394192Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:45.358125925Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:45.35892695Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:45.719106623Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:45.719737168Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:45.837252029Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:45.837893884Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:46.374351379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:46.603712007Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:46.604044547Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:46.98591557Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:47.083118451Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:47.083441624Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:48.259875862Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:48.521277266Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:48.522071422Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:48.560802592Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:49.203290023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:49.49333712Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:49.493587305Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:49.653336031Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:53:50.859452318Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:51.626816235Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:51.627517785Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:51.897650972Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:51.898159449Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:51.988591345Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:52.07357009Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:52.074076826Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:52.579322327Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:52.690996599Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:52.999870033Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:53.000255355Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:53.319564864Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:53.896564828Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:54.977013573Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:54.977211919Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:55.296158788Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:55.29658941Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:55.470614046Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:55.470851247Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:55.76250686Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:55.762959332Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:56.403215937Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:58.279080083Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:58.857397832Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:59.290441459Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:59.290733393Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:59.705962601Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:53:59.706225618Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:00.03521485Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:00.035377039Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:54:00.859439062Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:00.930151638Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:01.010252183Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:01.010751712Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:01.593213175Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:02.347482758Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:02.347658499Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:02.689279206Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:02.68996733Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:03.698333834Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:04.675103099Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:04.720260935Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:07.173273334Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:07.207219334Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:07.228042648Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:07.228308675Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:07.252738845Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:07.253528283Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:08.011938026Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:08.840272136Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:09.381936951Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:09.382262322Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:09.691441058Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:09.691931295Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:10.010375846Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:10.431297924Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:54:10.859436045Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:12.357442841Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:12.358032681Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:12.596043203Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:14.840762675Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:16.844579991Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:16.845075973Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:17.043708268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:17.443082337Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:17.443409907Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:18.419936654Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:19.408436596Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:19.689743693Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:19.690236656Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:54:20.859500782Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:21.207439972Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:21.207796208Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:21.92079673Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:21.920968933Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:22.764930083Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:23.134879519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:23.13515056Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:25.697924174Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:25.726491691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:25.7267639Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:26.303294262Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:26.303590782Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:28.161526313Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:28.162211652Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:28.708486771Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:28.709193477Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:28.913909677Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:28.914263717Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:28.939587437Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:54:30.859745956Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:31.638520312Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:31.639418354Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:31.786758288Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:31.786963968Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:32.274348551Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:32.274743613Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:32.354999986Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:32.35547549Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:32.592996547Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:32.59345096Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:32.905499503Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:33.196306158Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:33.196648254Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:33.795738024Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:33.796737837Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:33.82740546Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:33.829027615Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:33.831967782Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:34.150874142Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:34.151143239Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:35.309016135Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:35.57729046Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:35.577790091Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:36.599037095Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:36.599432298Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:36.920928747Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:37.161118365Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:37.1615219Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:39.24034116Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:39.240748677Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:40.710403238Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:40.710707625Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:54:40.85943147Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:41.255463051Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:41.255832752Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:41.487200372Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:41.526521147Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:41.527092949Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:41.559098881Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:41.55939505Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:41.773486026Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:42.545866834Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:42.546314629Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:43.86731209Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:43.997169355Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:44.337791506Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:45.114235698Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:45.114518629Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:45.358149441Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:45.358949879Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:45.71624804Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:45.716939819Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:45.828780547Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:45.829403023Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:46.375546986Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:46.608308903Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:46.60869423Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:46.983718311Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:47.074850995Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:47.075227928Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:48.266488729Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:48.510864315Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:48.51132268Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:48.554956606Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:49.2015878Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:49.493701574Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:49.493963257Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:49.633978571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:54:50.85984289Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:51.628890767Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:51.629578291Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:51.894743055Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:51.895369183Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:51.981722707Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:52.077594387Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:52.077992447Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:52.578169251Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:52.690844462Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:52.998888699Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:52.999331218Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:53.329492476Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:53.920897809Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:54.981265395Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:54.981487465Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:55.296199844Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:55.296655076Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:55.471982047Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:55.472143084Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:55.736483726Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:55.736939234Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:56.402799326Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:58.284465897Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:58.854380069Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:59.278199906Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:59.278567675Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:59.70559203Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:59.705877425Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:59.869275508Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:54:59.869435052Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:55:00.859421079Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:00.930787825Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:01.053592903Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:01.054127494Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:01.568025678Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:02.392788963Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:02.429046447Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:03.116917601Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:03.118011975Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:03.700073743Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:04.689019136Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:04.721626763Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:07.174010147Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:07.203782777Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:07.222656268Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:07.222894386Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:07.288098664Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:07.288920976Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:08.012295072Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:09.464628478Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:09.465274956Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:09.722378558Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:09.72317529Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:09.786105797Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:10.040566916Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:10.430595113Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:55:10.859495725Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:12.365872378Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:12.366635833Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:12.520565362Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:14.843856455Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:16.842785591Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:16.843443245Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:17.079500206Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:17.354465676Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:17.354824928Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:18.392146413Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:19.406852937Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:19.709237513Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:19.709773769Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:55:20.859434762Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:21.249058446Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:21.249395785Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:21.93157146Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:21.931760763Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:22.75825884Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:23.14832497Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:23.148603292Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:25.702412408Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:25.727584228Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:25.727864522Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:26.272062591Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:26.272356716Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:27.379238969Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:27.379552789Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:28.159523598Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:28.160242369Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:28.74429885Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:28.745071326Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:28.935821279Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:55:30.859426857Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:31.65219606Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:31.652954895Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:31.788146006Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:31.788403525Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:31.959189514Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:31.959687039Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:32.356000435Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:32.356472682Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:32.592114255Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:32.592729201Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:32.905540679Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:33.196397377Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:33.196733794Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:33.795250606Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:33.795931657Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:33.81682671Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:33.817941269Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:33.820099697Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:34.17528143Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:34.175544497Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:35.31358953Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:35.577341328Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:35.577958096Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:36.643386163Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:36.644010477Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:36.920698632Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:37.161299485Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:37.161741443Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:39.127021992Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:39.12749395Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:40.685519084Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:40.68575416Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:55:40.859484775Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:41.247275702Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:41.247564897Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:41.48730338Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:41.517654075Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:41.518060848Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:41.557764532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:41.558068775Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:41.715398744Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:42.542924375Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:42.543377394Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:43.863589812Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:43.997993078Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:44.272332944Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:45.169861227Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:45.170146107Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:45.358131952Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:45.358988206Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:45.731825488Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:45.73257951Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:45.829181826Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:45.829802093Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:46.375658163Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:46.771964492Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:46.772513666Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:46.98389897Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:47.231803177Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:47.232129026Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:48.257842152Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:48.552979622Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:48.65099119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:48.651817705Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:49.201712998Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:49.557058719Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:49.557328216Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:49.732343258Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:55:50.861628534Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:51.61511545Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:51.615810602Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:51.925581602Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:51.926253879Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:51.986959311Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:52.099529018Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:52.1001044Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:52.582519955Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:52.690855138Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:52.999502065Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:52.999992913Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:53.300401206Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:53.89648055Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:54.978388117Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:54.978828848Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:55.296954228Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:55.297552508Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:55.471979514Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:55.472190312Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:55.644299888Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:55.644681389Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:56.407644454Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:58.254582038Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:58.852437278Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:59.277276876Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:59.277695036Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:59.706553977Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:59.706803611Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:59.886846359Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:55:59.88700926Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:56:00.859436957Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:00.930441562Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:01.022276999Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:01.022988067Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:01.641688904Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:02.344711129Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:02.344862403Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:02.675865097Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:02.676517307Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:03.699369409Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:04.721464086Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:04.776211954Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:07.210570088Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:07.247658002Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:07.25913149Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:07.259893173Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:07.267021842Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:07.267263832Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:08.012803129Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:09.115022251Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:09.435331648Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:09.436176812Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:09.676472153Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:09.677295486Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:10.036255848Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:10.433257034Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:56:10.859431424Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:12.345603607Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:12.346395569Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:12.520829146Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:14.840783335Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:16.715557417Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:16.716019526Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:17.07730766Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:17.412160558Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:17.412546726Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:18.173587523Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:19.408944828Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:20.004470184Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:20.005009007Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:56:20.859405079Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:21.206928253Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:21.207286044Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:21.919816447Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:21.920010182Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:22.774101779Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:23.135207478Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:23.135495395Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:25.698384474Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:25.727608351Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:25.727850759Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:26.286198755Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:26.286566381Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:27.380482023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:27.380810689Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:28.159996201Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:28.16054751Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:28.750212821Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:28.750795885Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:28.931535243Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:56:30.859410191Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:31.789343681Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:31.789627712Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:31.955413381Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:31.955829694Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:32.348221077Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:32.348610935Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:32.540125944Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:32.540761071Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:32.592709678Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:32.593176898Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:32.905768348Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:33.196256786Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:33.196530084Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:33.795336053Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:33.796141888Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:33.816442031Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:33.817643497Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:33.819841874Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:34.158156185Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:34.158400214Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:35.310149119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:35.587691686Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:35.588140911Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:36.606216291Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:36.606624688Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:36.923238285Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:37.161010095Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:37.161567621Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:39.247813281Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:39.270199204Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:40.7011016Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:40.701317212Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:56:40.859442713Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:41.258516961Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:41.258886033Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:41.491730124Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:41.520098362Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:41.52050763Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:41.56875331Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:41.569506773Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:41.717045111Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:42.552352531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:42.552869752Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:43.861477842Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:43.999774165Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:44.342157498Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:45.126575315Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:45.126933774Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:45.358195521Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:45.358925944Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:45.732707495Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:45.733408318Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:45.835913596Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:45.836526686Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:46.375270778Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:46.603824304Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:46.604187798Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:46.982927202Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:47.214389974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:47.214712751Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:48.257411215Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:48.553411062Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:48.653071891Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:48.653613604Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:49.215823043Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:49.555507826Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:49.555766509Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:49.634281932Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:56:50.859442494Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:51.614192301Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:51.614881527Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:51.913457232Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:51.914125962Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:51.982910924Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:52.09858678Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:52.099117521Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:52.577751571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:52.690563438Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:52.999875744Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:53.000464231Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:53.350056507Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:53.897433589Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:54.974103006Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:54.974315494Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:55.315149024Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:55.315619099Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:55.476786113Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:55.476970303Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:55.645454481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:55.645897834Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:56.408238212Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:58.269598898Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:58.854504628Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:59.180935008Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:59.181262196Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:59.710862499Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:59.711144271Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:59.885985475Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:56:59.886139434Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:57:00.859423634Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:00.930287088Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:01.143131737Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:01.143614491Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:01.989433411Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:02.411709538Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:02.41189464Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:02.687833948Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:02.688459234Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:03.697420612Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:04.678517082Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:04.721566241Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:07.181405422Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:07.210867255Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:07.229171474Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:07.229366185Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:07.438551807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:07.439341234Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:08.020302308Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:08.217341871Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:09.406386923Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:09.406952875Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:09.687769199Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:09.688480601Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:10.040441992Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:10.431312686Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:57:10.859458425Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:12.344262317Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:12.344843178Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:12.595292072Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:14.843499083Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:16.766045638Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:16.78356043Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:17.042876861Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:17.4155945Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:17.415920157Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:18.169888769Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:19.406444597Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:19.682204028Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:19.682793049Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:57:20.859459411Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:21.218856026Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:21.219187857Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:21.936025887Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:21.936353551Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:22.773134492Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:23.13497757Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:23.135224945Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:25.701113032Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:25.73273709Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:25.733186995Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:26.272126077Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:26.272400269Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:27.311035197Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:27.311293532Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:28.159079393Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:28.159549892Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:28.810937223Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:28.811388377Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:28.932036627Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:57:30.859420771Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:31.786901889Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:31.787139931Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:31.96882735Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:31.969221467Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:32.458097544Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:32.458425133Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:32.515482039Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:32.516048444Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:32.592296885Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:32.592799342Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:32.90554926Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:33.196402962Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:33.196753903Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:33.795260177Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:33.796027034Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:33.817707586Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:33.819049741Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:33.821581884Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:34.161148486Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:34.161392445Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:35.311650069Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:35.577145893Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:35.577604039Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:36.630381059Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:36.6309549Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:36.920593666Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:37.16087348Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:37.161330115Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:39.102970971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:39.103307913Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:40.72942509Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:40.729887769Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:57:40.859442821Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:41.398845755Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:41.399171413Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:41.487609478Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:41.521563444Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:41.525292168Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:41.555280775Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:41.555585824Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:41.723572435Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:42.549647143Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:42.55011081Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:43.953068768Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:44.000356823Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:44.339300319Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:44.571597367Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:44.571901496Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:45.358099537Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:45.358850624Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:45.726270536Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:45.726999935Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:45.838511348Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:45.839127259Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:46.385165526Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:46.613509347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:46.613896396Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:46.98559376Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:47.221572053Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:47.221920098Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:48.25501306Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:48.554186791Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:48.646984505Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:48.647553791Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:49.201442629Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:49.554833598Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:49.55506117Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:49.640543303Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:57:50.859438313Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:51.628252852Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:51.628986301Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:51.891456972Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:51.892068498Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:51.982957601Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:52.093044619Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:52.09352848Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:52.577753831Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:52.698332797Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:52.998980396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:52.999428556Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:53.329998905Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:53.898962447Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:54.970905747Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:54.971127662Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:55.333104569Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:55.333546075Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:55.412726602Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:55.412865589Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:55.753724732Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:55.754153872Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:56.404756211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:58.254223751Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:58.86481767Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:59.102967407Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:59.103338851Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:59.70685704Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:59.707154699Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:59.900261778Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:57:59.900412479Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:58:00.859428412Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:00.931625987Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:01.088932132Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:01.089431456Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:02.353545338Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:02.515493979Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:02.516431091Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:02.676475022Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:02.677276795Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:03.69996206Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:04.676346168Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:04.721837504Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:07.182324301Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:07.214470836Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:07.234092865Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:07.234331497Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:07.250208815Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:07.25099112Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:08.017094777Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:08.479559433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:08.866563364Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:08.867334307Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:09.687384707Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:09.688201606Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:10.027120728Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:10.433085646Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:58:10.859503007Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:12.404618361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:12.405208814Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:12.517285648Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:14.851608668Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:16.848067897Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:16.848701426Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:17.080773469Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:17.460515385Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:17.460933401Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:18.269903681Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:19.413643409Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:19.689176908Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:19.689694665Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:58:20.85942626Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:21.218289974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:21.218628728Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:21.90848158Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:21.90862734Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:22.758211444Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:23.13460379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:23.134861247Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:25.6972154Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:25.726337027Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:25.726604749Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:26.272868577Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:26.273122841Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:27.389996759Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:27.390308473Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:28.163485807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:28.164133117Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:28.530745128Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:28.531165571Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:29.046213196Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:58:30.859452998Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:31.787479575Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:31.787745035Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:32.279162113Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:32.279556898Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:32.447212222Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:32.447593721Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:32.499932764Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:32.500470215Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:32.564707134Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:32.56518841Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:32.905516426Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:33.198433428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:33.198955663Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:33.795273764Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:33.79601889Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:33.815751068Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:33.816917196Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:33.819134374Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:34.15039695Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:34.150657598Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:35.314198924Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:35.595766057Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:35.596242451Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:36.621496373Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:36.622006438Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:36.927635277Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:37.168303184Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:37.168773349Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:39.212051211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:39.212709161Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:40.690557662Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:40.691154675Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:58:40.860106329Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:41.263183034Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:41.263437783Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:41.431866297Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:41.432235173Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:41.488406732Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:41.557745145Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:41.558075926Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:41.717739602Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:42.549537204Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:42.549987092Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:43.862699498Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:44.001025425Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:44.272060879Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:45.122564866Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:45.122903459Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:45.358101736Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:45.358911189Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:45.715158946Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:45.715795723Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:45.832788529Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:45.833449753Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:46.374493372Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:46.61313327Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:46.61342977Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:46.983651472Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:47.069516466Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:47.069872305Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:48.265918525Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:48.578835656Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:48.658354818Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:48.659035518Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:49.20728441Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:49.555149977Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:49.555413034Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:49.638255356Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:58:50.859416647Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:51.627029907Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:51.627802892Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:51.927214808Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:51.927767114Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:51.982510586Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:52.071255015Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:52.071642772Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:52.578541721Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:52.691692546Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:52.998884081Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:52.999333479Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:53.326779438Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:53.897103689Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:54.980002279Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:54.980216727Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:55.337880864Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:55.338313723Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:55.416796105Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:55.416951878Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:55.758004359Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:55.758379449Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:56.403183646Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:58.262520906Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:58.884458444Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:59.304245545Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:59.304555896Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:59.705293485Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:59.705559933Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:59.900848389Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:58:59.901025911Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:59:00.859432621Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:00.930190274Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:01.107019273Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:01.107653623Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:01.61911392Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:02.391479463Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:02.391658849Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:02.706347443Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:02.707245392Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:03.697645577Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:04.686647177Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:04.720329778Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:07.18043902Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:07.214289056Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:07.23791235Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:07.238199413Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:07.259079822Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:07.259961511Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:08.022798103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:08.056242752Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:09.67617187Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:09.67656609Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:09.795177489Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:09.795621492Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:10.172355458Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:10.432245154Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:59:10.859522864Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:12.444964027Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:12.445536548Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:12.595823419Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:14.945476533Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:16.84511822Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:16.845575566Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:17.074753045Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:17.465323655Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:17.465914321Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:18.693089701Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:19.4080023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:20.005911974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:20.006428784Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:59:20.859430223Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:21.220892094Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:21.221272605Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:21.900203158Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:21.900409305Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:22.758176479Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:23.13487631Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:23.135201018Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:25.698049723Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:25.726518961Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:25.726779604Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:26.277596596Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:26.278162989Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:27.386387282Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:27.386946167Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:28.162383705Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:28.163246382Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:28.564439095Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:28.565167548Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:28.935089401Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:59:30.85941955Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:31.787368191Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:31.787682556Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:32.276215908Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:32.276587645Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:32.456229149Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:32.456578598Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:32.543509906Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:32.544083347Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:32.592333155Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:32.592770785Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:32.90555922Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:33.208715037Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:33.209030175Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:33.79535856Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:33.796146832Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:33.816350177Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:33.81747164Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:33.819741035Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:34.160057773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:34.16049865Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:35.310761608Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:35.576979601Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:35.577425155Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:36.632466979Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:36.632986162Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:36.923166698Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:37.161081111Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:37.161519395Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:39.102590684Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:39.103061353Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:40.7011257Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:40.701376869Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:59:40.871739115Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:41.329874186Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:41.330279004Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:41.492556933Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:41.689359646Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:41.699697348Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:41.711724022Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:41.712147048Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:41.751484989Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:42.383719577Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:42.384250995Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:43.855340055Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:43.998652453Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:44.337939158Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:45.13202283Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:45.132356777Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:45.358076048Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:45.358858013Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:45.713818114Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:45.714513874Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:45.848443346Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:45.849092518Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:46.374896583Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:46.601869685Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:46.602209042Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:46.980628447Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:47.074917336Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:47.075315288Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:48.255017154Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:48.554385088Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:48.647506618Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:48.648321252Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:49.201341413Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:49.493248714Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:49.49349306Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:49.653712363Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T05:59:50.85944022Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:51.615885117Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:51.616599681Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:51.916585886Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:51.917056994Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:51.982190009Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:52.099376291Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:52.0998154Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:52.576829381Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:52.69117885Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:52.998839443Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:52.999269708Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:53.339608678Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:53.895959313Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:54.976991004Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:54.97719802Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:55.331920316Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:55.332342788Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:55.430891553Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:55.431046831Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:55.824096849Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:55.824548638Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:56.405561531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:58.262481233Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:58.851651737Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:59.277516284Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:59.277930487Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:59.705635149Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:59.705883999Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:59.908842167Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T05:59:59.909009356Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:00:00.859453116Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:00.930418854Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:01.079275657Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:01.079801788Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:01.747444172Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:03.446593852Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:03.485489501Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:04.130444902Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:04.726198443Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:04.727147291Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:04.827546104Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:05.031490101Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:07.185639184Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:07.217039464Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:07.23737631Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:07.237652876Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:07.401648722Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:07.402369962Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:08.023902139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:09.755205264Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:09.755742883Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:09.864102203Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:09.864415113Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:10.026481946Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:10.438219427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:00:10.864387406Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:11.119925822Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:12.444324523Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:12.445059689Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:12.523091036Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:14.841151268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:16.760739418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:16.761304287Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:17.087056902Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:17.087370803Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:17.172063398Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:18.979885192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:19.431261703Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:19.684109456Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:19.684701692Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:00:20.859429365Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:21.245260828Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:21.24564079Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:21.917282727Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:21.917457077Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:22.869153828Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:23.136986847Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:23.137313877Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:25.702953263Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:25.729331873Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:25.729545473Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:26.301759719Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:26.302208576Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:27.375245917Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:27.375536835Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:28.159524948Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:28.160338294Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:28.557226844Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:28.557772711Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:29.088092484Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:00:30.859370771Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:31.789910717Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:31.790541173Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:32.28974976Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:32.290156619Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:32.481488875Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:32.481950227Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:32.50198431Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:32.502425879Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:32.593146096Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:32.593569868Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:32.905609969Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:33.197890994Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:33.198264144Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:33.795257855Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:33.796505323Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:33.817344253Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:33.818574389Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:33.820943166Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:34.154943194Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:34.155204038Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:35.311229138Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:35.577202981Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:35.577734451Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:36.642562721Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:36.643125107Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:36.927227376Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:37.161193036Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:37.161632084Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:39.468893314Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:39.469398877Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:40.710150739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:40.71101265Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:00:40.859509631Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:41.253170866Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:41.253575072Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:41.487353669Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:41.555643828Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:41.556402333Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:41.584070328Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:41.584381746Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:41.708728537Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:42.558373576Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:42.558869091Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:43.898421958Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:44.003449582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:44.274890051Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:45.156027971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:45.156339686Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:45.358068822Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:45.358878925Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:45.839083615Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:45.840117479Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:46.002629464Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:46.003357247Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:46.378252449Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:46.595826716Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:46.596192376Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:46.9842255Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:47.070324922Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:47.070622186Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:48.287429184Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:48.552531263Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:48.725182769Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:48.726257544Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:49.204447491Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:49.507814541Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:49.508057988Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:49.642697674Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:00:50.859449346Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:51.626782514Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:51.62745512Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:51.952986157Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:51.953712356Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:51.999536237Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:52.111182Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:52.111626096Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:52.578609104Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:52.693968341Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:52.999533535Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:52.999974026Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:53.34979357Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:53.897778143Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:55.311606148Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:55.312187238Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:55.340933874Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:55.341272064Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:55.491400859Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:55.491578121Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:55.899178456Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:55.89965374Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:56.404944807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:58.293638719Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:58.869977283Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:59.176491974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:59.17699503Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:59.70547368Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:59.705765825Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:59.890103647Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:00:59.89028018Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:01:00.859431396Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:00.932803756Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:00.993411951Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:00.994026254Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:01.916801899Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:02.34783481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:02.347970847Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:02.701719492Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:02.70242825Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:03.702454494Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:04.682786136Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:04.768861848Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:07.18748191Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:07.225912661Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:07.245946481Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:07.246179421Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:07.283410433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:07.284275498Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:08.01146034Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:09.647719035Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:09.648288539Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:09.772343839Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:09.772970782Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:10.041656301Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:10.436551286Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:10.790451948Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:01:10.870741766Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:12.482040815Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:12.482626928Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:12.59634059Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:14.887834326Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:16.725564066Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:16.726001401Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:17.079190618Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:17.405962613Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:17.406288596Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:18.198567517Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:19.418728741Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:19.698196473Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:19.698757153Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:01:20.859443989Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:21.962356831Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:21.96280634Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:21.979617266Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:21.979779981Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:22.758152754Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:23.135492825Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:23.136084622Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:25.697700537Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:25.727166674Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:25.727408723Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:26.293974105Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:26.294243854Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:27.37952461Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:27.380043802Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:28.160860926Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:28.161912665Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:28.870911554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:28.877216485Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:28.946393572Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:01:30.859441053Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:31.641401445Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:31.64197484Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:31.786947294Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:31.78719218Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:32.273804918Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:32.274277232Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:32.349752691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:32.350112738Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:32.593192926Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:32.593893894Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:32.905809265Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:33.19661607Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:33.196927696Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:33.795331294Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:33.796029561Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:33.816648953Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:33.817882687Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:33.820194076Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:34.150500164Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:34.150776795Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:35.31310726Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:35.577591388Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:35.578239526Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:36.699111203Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:36.699596515Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:36.920261596Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:37.161649602Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:37.162150883Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:39.254929584Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:39.25541113Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:40.726729469Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:40.733120089Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:01:40.859433408Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:41.283221911Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:41.283566451Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:41.486831358Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:41.528529475Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:41.5291568Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:41.611249691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:41.611730698Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:41.71810934Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:42.558776241Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:42.559221612Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:43.870932442Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:44.000059252Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:44.338423568Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:45.117828344Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:45.118148612Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:45.358130355Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:45.358896668Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:45.713196464Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:45.713898535Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:45.839338913Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:45.840312291Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:46.380346964Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:46.610034177Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:46.610368935Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:46.979588313Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:47.061449212Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:47.06176724Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:48.268215412Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:48.553728179Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:48.676402554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:48.677123928Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:49.20825203Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:49.559489154Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:49.559719652Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:49.646818576Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:01:50.859356542Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:51.627610916Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:51.628297707Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:51.903854543Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:51.904393191Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:51.988163327Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:52.074816017Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:52.0752019Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:52.577467945Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:52.690809759Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:52.999787725Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:53.000353361Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:53.332267459Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:53.897212093Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:54.974306246Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:54.974522503Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:55.295976951Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:55.296518489Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:55.434382684Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:55.434559683Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:55.752682155Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:55.753086879Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:56.403901292Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:58.259066339Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:58.851423682Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:59.343866783Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:59.344269838Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:59.706254646Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:59.706543297Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:59.872488459Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:01:59.872658522Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:02:00.859425252Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:00.930678732Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:00.942169327Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:00.942621737Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:01.772070312Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:02.345469725Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:02.345631782Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:02.695003019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:02.695706437Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:03.69708058Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:04.74017268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:04.77377572Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:07.186645501Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:07.223377489Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:07.246734663Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:07.247014923Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:07.336300927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:07.337438914Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:08.011854732Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:08.375535375Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:09.392966769Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:09.393409094Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:09.691400663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:09.691850742Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:10.01720125Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:10.454438985Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:02:10.86866544Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:12.431413987Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:12.431980378Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:12.595194533Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:14.84053305Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:16.84002743Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:16.840476973Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:16.840894091Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:16.84118607Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:17.081037538Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:18.230538566Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:19.405782556Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:19.907416644Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:19.907934324Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:02:20.859436313Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:21.219845986Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:21.220189003Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:21.915833425Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:21.915986148Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:22.777361415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:23.134713191Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:23.134942698Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:25.697863799Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:25.728006823Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:25.728282924Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:26.273053027Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:26.273332833Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:27.381528681Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:27.38181834Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:28.161801727Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:28.162394958Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:28.750923866Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:28.751524657Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:28.960190198Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:02:30.859441215Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:31.650765965Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:31.651476177Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:31.787523013Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:31.787868478Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:31.964793487Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:31.965274199Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:32.364033444Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:32.364432065Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:32.592219947Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:32.592726492Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:32.905533778Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:33.196772327Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:33.197174212Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:33.795256546Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:33.796006909Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:33.816542694Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:33.817837169Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:33.820363025Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:34.150301329Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:34.150541243Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:35.318680228Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:35.577077847Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:35.577536885Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:36.73618361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:36.736588674Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:36.96072802Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:37.169580532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:37.170025457Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:39.399197373Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:39.399726307Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:40.686943671Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:40.687254769Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:02:40.859449162Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:41.266711239Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:41.267044104Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:41.488064925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:41.509197619Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:41.509579096Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:41.557296145Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:41.558059046Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:41.718286803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:42.219267318Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:42.219855033Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:43.889852891Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:43.997901701Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:44.340313833Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:44.550561782Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:44.550880664Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:45.358204545Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:45.359037658Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:45.716085046Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:45.716826036Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:45.832901002Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:45.833480396Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:46.374452228Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:46.664832772Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:46.665165202Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:46.985590109Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:47.071138264Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:47.071505015Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:48.271253433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:48.520846257Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:48.521440418Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:48.554174226Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:49.200636733Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:49.558862675Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:49.559140234Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:49.646887596Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:02:50.859467999Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:51.640425674Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:51.641121496Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:51.890855754Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:51.891397111Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:51.983269137Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:52.082552546Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:52.083094309Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:52.578559271Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:52.691706507Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:53.002699218Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:53.003132016Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:53.336728708Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:53.897982439Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:55.036564488Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:55.036804666Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:55.295941395Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:55.296481269Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:55.412538395Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:55.412719509Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:55.818283478Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:55.818880508Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:56.406273052Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:58.264945512Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:58.851584039Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:59.302137493Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:59.302459358Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:59.705334513Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:59.705622369Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:59.870273314Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:02:59.870429338Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:03:00.860800705Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:00.930562225Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:01.119208062Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:01.119761324Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:02.3488017Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:02.349003834Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:02.677195711Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:02.678093146Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:02.707568401Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:03.70019193Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:04.680202602Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:04.720491534Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:07.185934104Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:07.216164753Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:07.235391614Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:07.235621046Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:07.266147583Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:07.267070225Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:08.016265354Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:08.85716359Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:09.402802937Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:09.403088419Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:09.695189574Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:09.695699252Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:10.039967118Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:10.450107225Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:03:10.877879098Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:12.441159529Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:12.441744451Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:12.519820075Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:14.844719925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:16.841169513Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:16.841618217Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:16.915582896Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:16.915886337Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:17.070723886Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:18.192445025Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:19.408571203Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:19.678385067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:19.678910355Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:03:20.859428818Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:21.208147196Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:21.208526561Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:21.911084498Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:21.911306715Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:22.77394019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:23.135930899Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:23.13622479Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:25.697362134Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:25.726545867Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:25.726793872Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:26.273346348Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:26.273790543Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:27.379305931Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:27.379585978Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:28.159215039Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:28.160072516Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:28.754993567Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:28.755521386Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:28.935360967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:03:30.860872539Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:31.734549132Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:31.735917622Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:31.790320313Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:31.791061378Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:32.017242036Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:32.018296152Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:32.396544263Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:32.39865661Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:32.601295288Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:32.601809563Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:32.905519219Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:33.19811127Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:33.198434152Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:33.795752134Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:33.796970632Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:33.826605346Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:33.828527161Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:33.831719025Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:34.15025815Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:34.150619068Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:35.322396716Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:35.576738512Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:35.577198031Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:36.626422451Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:36.627234042Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:36.95996961Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:37.165934767Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:37.166360652Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:39.157411606Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:39.157883129Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:40.686601922Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:40.686895076Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:03:40.85943047Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:41.251056128Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:41.251394089Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:41.488860639Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:41.510150691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:41.510515696Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:41.554185491Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:41.554452326Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:41.792936659Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:42.556988013Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:42.557434599Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:43.859706518Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:44.003171238Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:44.33894676Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:45.121951537Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:45.122353748Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:45.358099116Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:45.358845524Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:45.71687464Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:45.717797283Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:45.829721963Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:45.83052201Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:46.374556006Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:46.632169152Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:46.632624122Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:46.98175275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:47.083630474Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:47.084054804Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:48.260753235Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:48.511932532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:48.512507143Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:48.555333541Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:49.200092407Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:49.556491892Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:49.55674661Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:49.651826091Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:03:50.859443423Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:51.628743568Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:51.629446111Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:51.926335598Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:51.926901284Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:51.98828765Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:52.077246498Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:52.077792088Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:52.578023276Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:52.690823075Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:52.998875663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:52.999411177Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:53.346141383Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:53.896381789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:54.976135713Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:54.976322498Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:55.296089843Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:55.296585627Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:55.412392519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:55.412517644Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:55.637771304Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:55.638189254Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:56.402319021Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:58.286159139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:58.857260616Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:59.300312618Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:59.300616007Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:59.705764379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:59.706026661Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:59.869499918Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:03:59.869647196Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:04:00.859456081Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:00.930637613Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:01.053508943Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:01.053991613Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:01.855297428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:02.349716073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:02.349863911Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:02.676791169Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:02.677608735Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:03.699532421Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:04.676782279Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:04.726721778Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:07.234214338Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:07.309439435Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:07.322286605Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:07.340365833Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:07.367936686Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:07.368266509Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:08.053542828Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:08.997570957Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:09.618549685Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:09.619013928Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:09.700193971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:09.700809477Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:10.066007749Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:10.460243655Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:04:10.859437878Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:12.443003473Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:12.443904219Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:12.516427394Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:14.863164772Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:16.728122992Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:16.729047886Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:17.084132543Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:17.448710292Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:17.449133144Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:18.638484661Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:19.412917802Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:19.68370568Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:19.684236725Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:04:20.859441406Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:21.220628905Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:21.220970922Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:21.909309364Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:21.909488961Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:22.764596186Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:23.134681576Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:23.134899589Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:25.698377362Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:25.782558694Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:25.782920326Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:26.29304309Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:26.293346159Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:27.326935596Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:27.327240472Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:28.278181812Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:28.279033037Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:28.741700337Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:28.742197107Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:28.971464967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:04:30.859444452Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:31.619957653Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:31.620452921Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:31.795135303Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:31.795747711Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:32.276894576Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:32.277403893Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:32.517186119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:32.517573437Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:32.621533949Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:32.622068335Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:32.905786746Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:33.196453496Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:33.196781086Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:33.795269769Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:33.796058389Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:33.821932762Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:33.823138806Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:33.826091838Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:34.150603113Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:34.150860918Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:35.310106987Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:35.577293109Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:35.577837351Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:36.609992837Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:36.610421011Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:36.93546298Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:37.16100802Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:37.161448536Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:39.264192476Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:39.264525749Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:40.685767718Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:40.686023172Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:04:40.859518179Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:41.257954105Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:41.258229215Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:41.488891424Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:41.529215957Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:41.529685373Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:41.55746987Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:41.557808983Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:41.722478941Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:42.565155015Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:42.565617848Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:43.954753413Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:43.999774987Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:44.337890429Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:45.119975691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:45.120290453Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:45.358203325Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:45.359153286Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:45.714543732Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:45.715218034Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:45.842502025Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:45.843110403Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:46.378371677Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:46.608984829Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:46.609328614Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:46.983810046Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:47.076293415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:47.076831659Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:48.258302998Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:48.520021482Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:48.520460156Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:48.555076986Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:49.228827452Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:49.555083582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:49.555340531Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:49.640972717Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:04:50.859535706Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:51.622109253Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:51.622831827Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:51.906474136Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:51.907087469Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:51.983793492Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:52.087952541Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:52.088487333Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:52.578097589Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:52.692458167Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:52.998844236Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:52.999307715Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:53.301166803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:53.896557905Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:54.993048634Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:54.993273514Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:55.332435701Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:55.333282347Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:55.433127011Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:55.433641407Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:55.795195026Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:55.795846492Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:56.403746301Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:58.338345034Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:58.857747531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:59.313365205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:59.31378171Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:59.705810106Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:04:59.706116668Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:00.064433935Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:00.064599084Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:05:00.859439667Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:00.930705812Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:01.091011641Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:01.09151713Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:02.692750952Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:02.692997081Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:02.885309496Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:02.886412272Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:03.335460211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:03.698972497Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:04.685597765Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:04.722271855Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:07.2072691Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:07.243532747Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:07.274614309Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:07.275061061Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:07.312560672Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:07.313653003Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:08.018516652Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:09.412541624Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:09.413052829Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:09.71792126Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:09.718620472Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:10.032436118Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:10.526793249Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:05:10.859870694Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:11.020897765Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:12.438009741Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:12.438632837Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:12.515394914Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:14.840403809Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:16.714499325Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:16.714940672Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:17.050565704Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:17.452486967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:17.453109277Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:18.294920122Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:19.427476876Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:20.018246254Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:20.018919815Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:05:20.859412403Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:21.217989736Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:21.21845256Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:21.974573057Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:21.974774202Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:22.76022714Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:23.135356832Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:23.135658678Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:25.698822573Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:25.727278377Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:25.727556801Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:26.293527961Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:26.293883546Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:27.385527061Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:27.385870013Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:28.186240411Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:28.190716061Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:28.760791881Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:28.761324522Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:28.941370575Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:05:30.859427993Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:31.626007081Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:31.626594824Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:31.78738728Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:31.787649703Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:32.289962326Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:32.290376541Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:32.480357494Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:32.481308835Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:32.597723807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:32.598544605Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:32.922822511Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:33.200260358Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:33.200903535Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:33.840427187Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:33.841942666Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:33.954458556Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:33.963350078Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:33.968164273Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:34.158508926Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:34.162019042Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:35.311533767Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:35.578115806Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:35.578716125Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:36.62392043Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:36.624395313Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:36.929738566Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:37.160843901Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:37.161262821Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:39.257538345Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:39.258157949Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:40.711495995Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:40.711759321Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:05:40.859422136Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:41.076293532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:41.076612173Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:41.487845603Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:41.539121691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:41.539525764Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:41.572722904Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:41.573100313Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:41.721921828Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:42.563687389Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:42.564136122Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:43.858702536Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:44.002732437Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:44.272225925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:45.157735526Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:45.158054786Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:45.358102102Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:45.358887874Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:45.713491101Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:45.714175357Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:45.837485244Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:45.838107239Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:46.374711742Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:46.774825124Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:46.775184111Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:46.980619941Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:47.059752513Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:47.060087425Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:48.254164019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:48.553136043Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:48.675457727Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:48.676163401Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:49.200844873Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:49.493510765Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:49.493785638Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:49.64786504Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:05:50.859430326Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:51.615112258Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:51.615858726Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:51.898216205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:51.898950873Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:51.983533994Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:52.076615211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:52.07707286Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:52.579754821Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:52.692231408Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:53.000383182Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:53.000793969Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:53.304479462Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:53.896472667Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:54.977842966Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:54.978053532Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:55.333061455Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:55.333492159Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:55.475754907Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:55.475914382Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:55.75503841Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:55.755610232Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:56.40275348Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:58.28408115Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:58.87684671Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:59.301253241Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:59.301650016Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:59.705691135Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:59.705979316Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:59.887060859Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:05:59.88721677Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:06:00.860334772Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:00.931230248Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:01.100428043Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:01.100971775Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:02.306647102Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:02.406843935Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:02.407025177Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:02.780462705Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:02.781191848Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:03.699426876Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:04.678319446Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:04.720947585Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:07.18557932Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:07.220958753Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:07.247411688Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:07.247722195Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:07.272184715Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:07.272909963Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:08.013886071Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:08.896647451Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:08.897046774Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:09.217024778Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:09.68213901Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:09.682596425Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:10.011712052Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:10.432820229Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:06:10.859463094Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:12.469819618Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:12.470615371Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:12.578015221Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:14.843726859Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:16.842256983Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:16.842845328Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:17.079143347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:17.461904555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:17.46230358Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:18.477279191Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:19.405833322Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:19.781207172Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:19.782053418Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:06:20.859430663Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:21.222229099Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:21.222569128Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:21.944102273Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:21.944278799Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:22.774299653Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:23.175168369Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:23.175424478Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:25.697313638Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:25.727033375Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:25.727266104Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:26.303367393Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:26.30377969Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:27.38142283Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:27.381806485Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:28.161489509Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:28.162448045Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:28.591924158Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:28.596083983Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:28.939483064Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:06:30.859416732Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:31.640970876Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:31.64179136Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:31.804276512Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:31.804786062Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:32.289570233Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:32.290011284Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:32.467202995Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:32.467606248Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:32.593486596Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:32.594228733Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:32.905560141Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:33.197525103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:33.197898214Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:33.795377802Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:33.796219154Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:33.820269679Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:33.821541784Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:33.824125514Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:34.152815588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:34.153142409Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:35.310901065Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:35.578169124Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:35.578758891Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:36.611792663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:36.612263253Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:36.922074393Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:37.162432395Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:37.163008719Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:39.294452354Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:39.295159546Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:40.767079479Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:40.767451803Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:06:40.898810927Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:41.247214807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:41.247547189Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:41.491371561Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:41.526550866Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:41.527077435Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:41.571143395Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:41.571654306Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:41.721290118Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:42.55478105Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:42.555219525Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:43.860778457Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:43.997701431Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:44.272031766Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:45.123017285Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:45.123412679Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:45.358220001Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:45.35897083Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:45.727754069Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:45.728418845Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:45.830897715Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:45.83151063Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:46.374301403Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:46.608819572Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:46.609158848Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:46.98537206Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:47.07576979Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:47.076125378Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:48.26231513Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:48.554313507Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:48.647412184Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:48.648166386Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:49.213651356Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:49.50390046Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:49.504121207Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:49.653427985Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:06:50.859448573Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:51.673406546Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:51.674128599Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:51.914149648Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:51.915005928Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:51.982261185Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:52.077879786Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:52.078302487Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:52.578573762Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:52.691868857Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:52.998542086Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:52.999000119Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:53.301643288Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:53.897802926Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:54.981306333Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:54.981524474Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:55.335391758Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:55.336586797Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:55.475271441Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:55.475461911Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:55.812442379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:55.812917404Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:56.402513379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:58.304955382Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:58.8674534Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:59.137224251Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:59.137637397Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:59.706228451Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:59.706494505Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:59.885653745Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:06:59.885831194Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:07:00.859506981Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:00.933249379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:00.978561316Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:00.979200321Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:01.505061049Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:02.406656582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:02.406884574Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:02.690403317Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:02.691333417Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:03.698155264Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:04.720966671Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:04.772127286Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:07.184391131Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:07.217939239Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:07.237969332Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:07.238203373Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:07.256132913Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:07.256834972Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:08.018995467Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:08.30719191Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:08.820614591Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:08.821028539Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:09.682449205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:09.683342602Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:10.01577992Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:10.430834897Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:07:10.85942798Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:12.441108941Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:12.441737023Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:12.598483954Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:14.840424806Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:16.840348313Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:16.841455447Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:17.074277268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:17.403346416Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:17.403696439Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:18.153897276Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:19.405965966Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:19.679301669Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:19.679853688Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:07:20.859528491Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:21.210913671Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:21.211270439Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:21.904275999Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:21.904441561Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:22.757518614Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:23.13456071Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:23.134862069Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:25.69846648Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:25.726437915Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:25.726683974Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:26.296344906Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:26.296617587Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:27.326801859Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:27.327074566Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:28.174846296Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:28.175408228Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:28.575034959Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:28.57556357Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:28.945523657Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:07:30.859443994Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:31.629562634Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:31.630160278Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:31.788012245Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:31.788349055Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:32.271920605Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:32.272315252Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:32.482810636Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:32.483206458Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:32.575200777Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:32.575698031Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:32.905578828Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:33.197868535Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:33.198274317Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:33.795814691Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:33.796617028Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:33.82313447Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:33.824542821Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:33.826764461Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:34.150266104Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:34.150536555Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:35.317169127Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:35.614931734Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:35.618305473Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:36.684774074Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:36.685206101Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:36.923802333Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:37.162477943Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:37.162965628Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:39.096918152Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:39.097390577Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:40.692255467Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:40.692589228Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:07:40.859493226Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:41.25278Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:41.253099515Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:41.488723472Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:41.515993206Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:41.516329111Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:41.556101558Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:41.556427117Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:41.791190708Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:42.552292811Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:42.552798977Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:43.95445625Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:43.998422404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:44.340969924Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:44.565080785Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:44.565432384Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:45.358134605Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:45.358939006Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:45.712772083Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:45.71345736Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:45.829774888Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:45.830370387Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:46.377282596Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:46.603533278Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:46.603874532Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:46.985844176Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:47.062384527Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:47.062708435Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:48.258100397Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:48.553312416Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:48.64857943Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:48.649317339Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:49.201158702Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:49.504126639Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:49.504383814Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:49.644102516Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:07:50.859430549Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:51.629227874Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:51.629945822Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:51.903551476Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:51.904097406Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:51.980018283Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:52.074150234Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:52.074757445Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:52.578318423Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:52.690741625Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:52.999180006Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:52.999654346Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:53.310774177Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:53.897568279Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:54.986953772Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:54.98719905Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:55.34237638Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:55.342844266Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:55.47467852Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:55.474832097Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:55.747964722Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:55.748385683Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:56.40184353Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:58.300092582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:58.861497271Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:59.094694999Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:59.094990445Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:59.709248457Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:59.709506788Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:59.886791479Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:07:59.886940096Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:08:00.859500292Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:00.930362523Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:00.984620502Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:00.985124724Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:01.643234583Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:02.363413101Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:02.363576537Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:02.683162012Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:02.684023119Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:03.700694832Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:04.678442173Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:04.721110589Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:07.181522836Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:07.212177731Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:07.235465746Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:07.235790798Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:07.270941541Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:07.271696987Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:08.015091761Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:08.343391652Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:09.707044085Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:09.707626184Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:09.863998878Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:09.86439523Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:10.063790723Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:10.432916106Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:08:10.859439516Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:12.445443264Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:12.446179032Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:12.516897484Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:14.841377153Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:16.84872635Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:16.849178217Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:17.043236397Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:17.40299389Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:17.403384929Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:18.1524077Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:19.412251466Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:20.001049385Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:20.001567916Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:08:20.859507221Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:21.207213976Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:21.207544393Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:21.909705923Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:21.909874445Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:22.77353484Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:23.135465559Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:23.135759389Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:25.697380075Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:25.726360236Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:25.726589013Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:26.343250188Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:26.343540127Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:27.378628575Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:27.37892396Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:28.18146389Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:28.184004028Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:28.518731142Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:28.519262869Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:29.030113611Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:08:30.859441682Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:31.786827177Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:31.787088817Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:31.97399059Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:31.974372852Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:32.351837017Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:32.352198994Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:32.457157793Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:32.457655337Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:32.5922522Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:32.592742633Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:32.905601219Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:33.196915038Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:33.197234701Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:33.795287247Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:33.796028566Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:33.816236831Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:33.817610372Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:33.819903736Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:34.162746463Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:34.162995317Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:35.310788131Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:35.577963904Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:35.578454011Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:36.618679602Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:36.619069526Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:36.921063012Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:37.166923123Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:37.167348697Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:39.243069722Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:39.243503693Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:40.700961679Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:40.701224592Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:08:40.859444654Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:41.233222752Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:41.233480577Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:41.427755822Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:41.428145407Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:41.487100807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:41.555261809Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:41.555562228Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:41.709874609Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:42.555499451Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:42.556242076Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:43.953756213Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:43.998117961Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:44.337996846Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:45.119127323Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:45.119403583Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:45.358082087Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:45.358859977Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:45.722724327Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:45.723428965Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:45.837380413Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:45.83796334Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:46.377085935Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:46.591232377Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:46.591748555Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:46.985100367Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:47.214640379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:47.214991824Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:48.260994225Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:48.555681685Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:48.647000984Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:48.648075913Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:49.202240493Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:49.556345676Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:49.556547674Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:49.73107153Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:08:50.859432181Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:51.627552266Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:51.628231833Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:51.913539251Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:51.914205869Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:51.980190959Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:52.075675361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:52.076095344Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:52.578596648Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:52.691696167Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:52.99884147Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:52.99935084Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:53.32211237Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:53.897264539Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:54.982382552Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:54.982597382Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:55.295994691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:55.296463141Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:55.476404039Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:55.476587614Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:55.748274496Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:55.748733361Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:56.423108769Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:58.29257933Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:58.8520744Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:59.287724482Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:59.288033166Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:59.705411891Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:59.705704772Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:59.890267789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:08:59.890421962Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:09:00.859509075Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:00.930523495Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:00.989307838Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:00.98978605Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:01.920378424Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:02.349406007Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:02.349563162Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:02.676207068Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:02.676953634Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:03.704655087Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:04.67586103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:04.720493351Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:07.18129535Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:07.213201568Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:07.240210167Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:07.240457212Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:07.266195601Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:07.266886817Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:08.012246218Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:08.144416146Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:09.685627386Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:09.686030314Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:09.709426749Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:09.710001957Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:10.015923995Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:10.440074448Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:09:10.859447736Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:12.43716453Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:12.437787848Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:12.515126995Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:14.862387892Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:16.714863201Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:16.715301514Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:17.170837081Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:17.422062387Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:17.422404981Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:18.311331522Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:19.407716725Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:19.678952955Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:19.679470401Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:09:20.859429397Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:21.212300517Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:21.212684792Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:21.925245344Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:21.925437717Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:22.773444347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:23.137174345Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:23.137426278Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:25.698003019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:25.726540055Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:25.726802413Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:26.272152631Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:26.272432372Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:27.380147703Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:27.380416122Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:28.171252164Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:28.171928712Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:28.628977657Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:28.62952204Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:28.931426818Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:09:30.859417222Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:31.786775691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:31.786997935Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:31.955688723Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:31.956089607Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:32.38206185Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:32.382532282Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:32.546102766Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:32.546565956Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:32.597418187Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:32.597909144Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:32.905474419Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:33.196880107Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:33.197213862Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:33.795350912Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:33.796061151Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:33.816066783Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:33.81722106Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:33.819371076Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:34.15081694Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:34.151059689Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:35.313024898Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:35.577030175Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:35.577454951Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:36.674365694Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:36.675539977Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:36.935294989Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:37.164597283Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:37.165491392Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:39.331216675Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:39.333000609Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:40.7008989Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:40.701162191Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:09:40.859426953Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:41.25469639Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:41.254988924Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:41.486537705Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:41.517988742Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:41.518382Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:41.562420747Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:41.562925361Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:41.715197775Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:42.402763042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:42.403191679Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:43.853970104Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:43.998011042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:44.338170872Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:44.857865372Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:44.858162314Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:45.358106767Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:45.358860155Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:45.716497594Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:45.717189384Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:45.830233833Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:45.830875247Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:46.374364842Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:46.622960183Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:46.623343029Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:46.985304301Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:47.225937201Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:47.226231289Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:48.256742814Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:48.554302266Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:48.646488582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:48.652379289Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:49.202463553Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:49.557026004Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:49.557263043Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:49.644601063Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:09:50.859433893Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:51.625264549Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:51.626005198Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:51.913686775Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:51.914293007Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:51.981120072Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:52.079888019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:52.080295061Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:52.576839953Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:52.690990363Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:52.999001304Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:52.999440461Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:53.300384665Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:53.897528466Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:54.971970212Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:54.972179799Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:55.296190296Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:55.296629312Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:55.478514266Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:55.478720342Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:55.639386546Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:55.639850001Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:56.402590534Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:58.285911162Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:58.85179175Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:59.287385072Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:59.287648698Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:59.705829278Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:59.706083121Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:59.887049892Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:09:59.887209182Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:10:00.859507963Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:00.930274121Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:01.151118265Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:01.151768649Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:03.396131595Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:03.713905926Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:03.814992431Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:03.815962466Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:03.871198963Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:04.065032511Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:04.682697145Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:04.757199456Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:07.343391597Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:07.344723787Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:07.359542241Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:07.412641915Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:07.454303924Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:07.454707385Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:08.028203041Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:09.012064907Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:09.478468196Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:09.479056081Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:09.927602051Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:09.928318106Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:10.029718747Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:10.457142301Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:10:10.859469307Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:12.441147897Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:12.441805743Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:12.597081945Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:14.864986211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:16.63388193Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:16.634249998Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:16.720099221Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:16.720651958Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:17.090155517Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:18.275028188Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:19.408850618Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:19.691278331Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:19.691805637Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:10:20.859440923Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:21.242636305Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:21.24301416Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:21.918312783Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:21.919078689Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:22.791403165Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:23.142652792Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:23.142996334Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:25.698048917Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:25.727339521Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:25.727567313Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:26.272130923Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:26.272411033Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:27.338203387Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:27.338459362Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:28.160546037Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:28.161506696Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:28.75129139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:28.751765151Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:28.984569853Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:10:30.859433182Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:31.787162492Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:31.787410105Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:32.273175399Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:32.273601186Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:32.35998467Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:32.360311121Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:32.490862673Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:32.491381592Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:32.564735104Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:32.565200365Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:32.905892425Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:33.265241077Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:33.265510123Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:33.795227969Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:33.796165329Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:33.817579161Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:33.818840582Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:33.821218774Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:34.150886503Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:34.151156066Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:35.312488099Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:35.586277286Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:35.586953587Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:36.621199501Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:36.621634154Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:36.926446373Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:37.164286714Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:37.164907346Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:39.378419774Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:39.378895842Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:40.686055347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:40.686310672Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:10:40.859427261Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:41.315918419Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:41.316256239Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:41.510542404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:41.594478768Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:41.594907729Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:41.60633675Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:41.606765081Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:41.740302789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:42.549638546Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:42.550069176Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:43.876000502Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:44.002097887Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:44.268989658Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:45.128184826Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:45.1285427Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:45.358097137Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:45.358856314Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:45.717973908Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:45.718709604Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:45.841868513Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:45.842470466Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:46.375638251Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:46.594131921Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:46.594484531Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:46.984970415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:47.06046451Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:47.060804099Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:48.307275725Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:48.553066948Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:48.656204164Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:48.675970317Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:49.225606244Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:49.557372001Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:49.558181548Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:49.637995299Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:10:50.859420627Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:51.619063953Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:51.619721711Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:51.90124826Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:51.901948313Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:52.080834331Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:52.11618921Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:52.116550379Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:52.578157808Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:52.692376622Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:52.998779596Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:52.999361751Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:53.300552375Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:53.902580148Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:54.973120484Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:54.973345716Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:55.487412631Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:55.487767565Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:55.543858929Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:55.544434187Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:55.664562444Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:55.66499222Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:56.402857277Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:58.258887946Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:58.851423803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:59.091485791Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:59.091958851Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:59.705933034Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:59.706225449Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:59.887197485Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:10:59.887355018Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:11:00.859438919Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:00.931154974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:01.052709529Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:01.053211377Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:01.569232426Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:02.346230835Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:02.346387645Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:02.691879026Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:02.69253468Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:03.698561214Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:04.681539171Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:04.722256Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:07.169322881Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:07.200642437Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:07.219516287Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:07.219757086Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:07.276084231Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:07.276851083Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:08.020128281Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:08.241253768Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:09.421234621Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:09.42163652Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:09.949152437Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:09.950063332Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:10.111527126Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:10.454501378Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:11:10.859429787Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:12.435549871Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:12.436169119Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:12.51563432Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:14.844094391Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:16.841582837Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:16.842066666Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:17.077138107Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:17.410627967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:17.410980172Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:18.244888205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:19.411125947Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:19.994937435Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:19.995432652Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:11:20.85941581Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:21.23476747Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:21.235227117Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:21.914765399Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:21.914951736Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:22.76061665Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:23.142963503Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:23.143285553Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:25.697889418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:25.727991569Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:25.728206029Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:26.347090602Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:26.347353488Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:28.196178348Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:28.197072959Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:28.764970301Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:28.765503771Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:28.931160264Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:11:30.859420575Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:31.652805362Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:31.653331681Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:31.788579899Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:31.78886669Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:32.2863983Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:32.28708151Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:32.475842588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:32.476210208Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:32.592459329Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:32.592931404Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:32.905497312Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:33.196431883Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:33.196766879Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:33.795345745Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:33.796076805Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:33.815990939Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:33.817120808Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:33.819308407Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:34.151014679Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:34.151287441Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:35.310557707Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:35.576828943Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:35.577258321Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:36.644201604Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:36.647160668Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:36.920202893Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:37.160909258Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:37.161361196Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:37.302632889Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:37.302933239Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:39.223928872Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:39.224401768Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:40.710959452Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:40.711449154Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:11:40.859985426Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:41.395918921Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:41.396739978Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:41.594582723Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:41.611321909Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:41.63449277Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:41.635324842Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:41.648168093Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:41.727000661Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:42.559339111Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:42.55975343Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:43.861511179Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:44.000687414Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:44.339086729Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:45.135736016Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:45.136040248Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:45.358351792Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:45.359093703Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:45.715992021Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:45.716628189Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:45.827285564Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:45.827883779Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:46.375281264Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:46.593980962Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:46.594322265Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:46.983034881Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:47.078163106Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:47.078500395Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:48.260771411Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:48.558583091Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:48.643003426Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:48.643427252Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:49.200536709Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:49.554492547Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:49.554788334Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:49.654011803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:11:50.859465245Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:51.630332829Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:51.631013295Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:51.89871915Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:51.899367931Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:51.984593627Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:52.065906625Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:52.066295152Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:52.577359164Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:52.702223236Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:52.999040522Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:52.999628059Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:53.301046595Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:53.896468608Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:54.998607856Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:54.998893188Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:55.33198156Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:55.332914219Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:55.475255649Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:55.475416421Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:55.740847235Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:55.741274635Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:56.403085312Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:58.280483765Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:58.851859236Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:59.317933355Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:59.318253414Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:59.70606403Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:59.706351756Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:59.870508742Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:11:59.870677624Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:12:00.859428419Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:00.930190328Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:00.98049753Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:00.980982208Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:02.112464484Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:02.422498278Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:02.42272845Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:02.7183265Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:02.719139014Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:03.698887022Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:04.67859281Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:04.721474668Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:07.188775931Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:07.239174348Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:07.272829695Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:07.273154212Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:07.325133534Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:07.326206244Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:08.011203278Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:08.616780676Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:09.413812212Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:09.41415492Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:09.712028325Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:09.712717915Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:10.06411722Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:10.4593016Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:12:10.859433826Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:12.450301666Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:12.451092387Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:12.519376436Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:14.846471955Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:16.603248041Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:16.603580474Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:16.842476324Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:16.84295575Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:17.076643412Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:18.108518702Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:19.409958762Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:19.691940215Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:19.692498406Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:12:20.859414933Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:21.215870962Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:21.216227925Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:21.930721887Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:21.930905518Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:22.758438518Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:23.134658719Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:23.134930735Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:25.697979883Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:25.726406071Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:25.726627016Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:26.272221528Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:26.272505722Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:27.383705475Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:27.383989562Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:28.165937092Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:28.166634152Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:28.716512731Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:28.716967912Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:29.027806484Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:12:30.859447576Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:31.629598468Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:31.630119654Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:31.787639755Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:31.787951868Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:32.277384342Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:32.277749829Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:32.446105579Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:32.446483702Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:32.592581876Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:32.59303287Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:32.905528386Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:33.196828989Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:33.19719877Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:33.795389952Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:33.796154002Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:33.81662341Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:33.817783214Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:33.81993359Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:34.150275588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:34.150523382Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:35.310484782Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:35.577104879Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:35.577591208Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:36.595392109Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:36.595788594Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:36.920932517Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:37.170222785Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:37.170708971Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:39.20859271Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:39.209014114Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:40.686045508Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:40.686313606Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:12:40.859490623Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:41.256832379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:41.257131155Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:41.489333758Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:41.513223242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:41.513599392Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:41.556820073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:41.557139142Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:41.726956583Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:42.22693873Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:42.2273556Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:43.87098553Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:43.998638377Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:44.338039376Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:44.561413218Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:44.561733923Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:45.358060167Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:45.358809269Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:45.719298673Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:45.720005877Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:45.828420594Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:45.829061264Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:46.376120521Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:46.630051032Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:46.630401602Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:46.985868995Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:47.060205015Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:47.060519417Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:48.276596697Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:48.553474799Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:48.606799342Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:48.607213716Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:49.233779689Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:49.494412638Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:49.494681536Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:49.636781959Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:12:50.859438666Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:51.61681552Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:51.617513551Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:51.891931353Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:51.892579172Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:51.980149313Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:52.078419452Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:52.078845723Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:52.578219582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:52.691604976Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:52.998763741Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:52.999214764Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:53.317397624Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:53.896629112Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:55.109079367Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:55.109283941Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:55.333035005Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:55.33346175Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:55.433033974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:55.433208701Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:55.754296898Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:55.754755637Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:56.404567053Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:58.256980362Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:58.859169571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:59.306204718Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:59.306512377Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:59.707783284Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:59.70813107Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:59.889505689Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:12:59.889685132Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:13:00.859432727Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:00.930521846Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:01.083724182Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:01.084235708Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:01.767188919Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:02.377828933Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:02.37797001Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:02.677635532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:02.678374839Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:03.700915253Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:04.68015886Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:04.720536358Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:07.179236798Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:07.214480137Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:07.234085641Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:07.234324638Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:07.300589378Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:07.30137873Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:08.039493382Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:08.216058099Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:09.471253523Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:09.471655639Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:09.70506663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:09.706339525Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:10.010563239Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:10.430935246Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:13:10.859436211Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:12.440603382Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:12.441231115Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:12.595447655Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:14.840543754Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:16.72482935Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:16.725224358Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:17.048732842Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:17.049046359Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:17.07760798Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:18.299975973Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:19.406857511Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:19.715280599Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:19.715822605Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:13:20.859465585Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:21.219728463Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:21.220099311Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:21.907238999Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:21.907370216Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:22.773919803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:23.142828498Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:23.143122697Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:25.69878178Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:25.726291266Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:25.726500788Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:26.272208224Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:26.272479172Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:27.38245681Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:27.382754486Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:28.177176298Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:28.177913034Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:28.933020616Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:29.142152534Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:29.14260421Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:13:30.859414629Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:31.626633347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:31.627329058Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:31.7870244Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:31.787258873Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:32.284567944Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:32.285047086Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:32.477145974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:32.477489262Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:32.592266011Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:32.592728922Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:32.905651119Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:33.19777059Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:33.198168767Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:33.795270137Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:33.796003503Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:33.815560848Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:33.81668281Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:33.818841447Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:34.150599587Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:34.150843366Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:35.313331422Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:35.57706488Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:35.577514029Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:36.68676773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:36.687235578Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:36.920763647Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:37.17063509Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:37.171119084Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:39.114796737Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:39.115160104Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:40.714198203Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:40.714678424Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:13:40.861765255Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:41.261404019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:41.261746905Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:41.487321435Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:41.506170396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:41.506508589Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:41.573268711Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:41.573578649Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:41.720387295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:42.617954646Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:42.619049307Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:43.857863878Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:43.99814433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:44.338753841Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:45.122920139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:45.123279971Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:45.3582111Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:45.359230295Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:45.713982311Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:45.714640495Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:45.839355157Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:45.840026618Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:46.379654332Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:46.680438939Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:46.680821637Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:46.986081773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:47.074384012Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:47.074767096Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:48.253444079Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:48.513038758Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:48.513656956Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:48.554969679Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:49.200322178Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:49.514064111Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:49.51429421Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:49.63658266Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:13:50.859497392Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:51.630985774Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:51.631711341Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:51.904253205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:51.904772773Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:51.985879305Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:52.077986192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:52.078422624Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:52.578974228Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:52.691413688Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:53.009071085Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:53.009518221Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:53.328881838Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:53.896966762Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:54.971974978Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:54.972189634Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:55.336786776Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:55.337211245Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:55.413870734Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:55.413990279Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:55.739639075Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:55.740120446Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:56.402456793Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:58.261648862Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:58.852357192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:59.287097163Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:59.287410706Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:59.706185975Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:13:59.706433203Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:00.063213645Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:00.063379766Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:14:00.859440685Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:00.931573342Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:00.999013737Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:00.999542427Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:01.937099255Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:02.367619803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:02.367808037Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:02.694485161Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:02.696538139Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:03.698709911Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:04.677631171Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:04.720390244Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:07.181249638Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:07.21242584Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:07.237367727Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:07.237681455Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:07.267657718Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:07.268351814Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:08.01600749Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:08.611918145Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:09.611541175Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:09.612356893Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:09.751826236Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:09.75261768Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:10.097933343Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:10.444579079Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:14:10.859448859Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:12.448794661Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:12.449575212Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:12.596276211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:14.841943994Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:16.715392771Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:16.715898056Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:17.093223095Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:17.415066118Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:17.415366117Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:18.642387978Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:19.415185054Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:20.009238329Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:20.009817567Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:14:20.859423355Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:21.254953292Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:21.255796661Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:21.940466726Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:21.941100335Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:22.760114553Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:23.134532829Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:23.134810209Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:25.697225427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:25.72646384Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:25.726748904Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:26.342203129Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:26.342493496Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:27.326603856Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:27.326883987Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:28.161346431Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:28.162080624Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:28.540366681Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:28.544796505Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:28.933635805Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:14:30.859440117Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:31.645247509Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:31.645810607Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:31.787211517Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:31.78746889Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:31.991455048Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:31.991925394Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:32.502140212Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:32.502553407Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:32.592823299Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:32.593359397Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:32.905584778Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:33.19642026Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:33.196756246Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:33.795219147Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:33.795996079Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:33.817138153Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:33.818275387Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:33.82046018Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:34.150137247Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:34.150387353Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:35.311898475Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:35.577855321Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:35.578344667Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:36.607363494Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:36.607877752Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:36.927627558Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:37.166206801Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:37.16662542Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:39.106328869Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:39.106847998Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:40.706576889Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:40.707211509Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:14:40.861743565Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:41.264061324Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:41.264440305Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:41.487842261Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:41.531857712Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:41.532255279Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:41.567057293Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:41.567405681Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:41.736517921Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:42.549471056Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:42.549936986Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:43.864458794Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:43.997566485Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:44.340159374Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:45.135570268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:45.135845345Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:45.358135841Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:45.358972562Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:45.718790161Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:45.71951007Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:45.832689647Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:45.833514603Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:46.374435667Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:46.651614889Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:46.65202356Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:46.981439807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:47.079335721Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:47.079696983Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:48.253738248Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:48.557135001Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:48.557397344Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:48.558081076Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:49.199790959Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:49.554379463Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:49.554630315Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:49.641679833Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:14:50.859427573Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:51.614063816Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:51.614808831Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:51.90117343Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:51.913226646Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:51.978838648Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:52.075365304Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:52.075780729Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:52.580219751Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:52.691088103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:52.999053909Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:52.99946679Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:53.328535215Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:53.897926152Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:55.031526401Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:55.031776915Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:55.332504784Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:55.33298938Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:55.412911041Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:55.41305249Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:55.739962258Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:55.740401588Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:56.430724282Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:58.259315831Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:58.857791569Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:59.293707311Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:59.29398396Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:59.706153902Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:59.706389687Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:59.869112122Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:14:59.869248992Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:15:00.859421432Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:00.930192724Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:01.014347002Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:01.014840131Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:01.516291469Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:02.440654829Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:02.440855361Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:02.926509179Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:02.927526212Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:03.699236747Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:04.739211855Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:04.776486269Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:07.184578361Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:07.250658826Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:07.279012015Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:07.279362677Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:07.315633995Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:07.316521421Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:08.011627542Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:08.822937507Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:08.823394285Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:09.160302612Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:09.693945686Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:09.697986617Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:10.047908861Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:10.436683092Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:15:10.859438182Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:12.45209027Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:12.452927113Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:12.51646613Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:14.8455001Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:16.846677923Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:16.84726365Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:17.0820186Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:17.406307756Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:17.406612698Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:18.483111102Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:19.408115586Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:19.677407045Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:19.677947165Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:15:20.859429577Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:21.2077065Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:21.208022825Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:21.928873314Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:21.929059965Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:22.777433935Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:23.136879833Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:23.137368458Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:25.698136977Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:25.772547597Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:25.772810355Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:26.272189934Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:26.272460686Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:27.381691138Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:27.382035594Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:28.229895226Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:28.230414013Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:28.576017621Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:28.576565506Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:28.953277575Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:15:30.859496678Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:31.644857324Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:31.645394172Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:31.791353938Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:31.79162127Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:31.955534576Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:31.956000015Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:32.364362082Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:32.364758Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:32.603644803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:32.604159364Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:32.905717574Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:33.196926138Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:33.197266703Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:33.795265904Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:33.797540823Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:33.817952896Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:33.819104352Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:33.821275336Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:34.150368687Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:34.150605003Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:35.311825149Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:35.57687596Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:35.577319619Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:36.639008505Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:36.639568442Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:36.920276487Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:37.161468786Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:37.161931941Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:39.092213588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:39.092726793Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:40.708836776Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:40.709181573Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:15:40.873828035Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:41.103116446Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:41.103415491Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:41.486836051Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:41.51869224Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:41.519074725Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:41.565540735Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:41.565922439Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:41.707797888Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:42.544112551Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:42.544691373Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:43.864121037Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:43.999491358Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:44.338489217Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:45.161736539Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:45.162105133Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:45.358106332Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:45.358860381Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:45.713344995Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:45.714046201Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:45.841349114Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:45.841936495Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:46.379895796Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:46.786899563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:46.787347662Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:46.983277712Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:47.059150076Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:47.059479459Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:48.258658631Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:48.553786873Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:48.641767867Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:48.642603599Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:49.236241108Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:49.554802811Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:49.555035333Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:49.649720416Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:15:50.859447697Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:51.621264303Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:51.622123311Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:51.8986556Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:51.899183271Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:52.081421147Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:52.084579182Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:52.084993343Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:52.577547263Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:52.694060489Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:52.998713404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:52.999159794Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:53.329512747Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:53.897146312Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:54.973621062Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:54.973832641Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:55.306174069Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:55.306645577Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:55.43018966Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:55.430349151Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:55.659444901Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:55.659828619Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:56.402332728Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:58.253963761Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:58.852509062Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:59.287089308Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:59.28739144Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:59.706404376Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:59.706747071Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:59.886593494Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:15:59.886757884Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:16:00.859423604Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:00.931876036Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:00.994449116Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:00.994977939Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:01.537516315Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:02.352505692Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:02.352927676Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:02.674907367Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:02.675692857Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:03.69753127Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:04.680693183Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:04.721517619Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:07.180275919Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:07.211565873Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:07.232372225Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:07.232637682Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:07.262378342Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:07.263136871Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:08.062054305Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:08.777981374Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:09.435231032Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:09.435526424Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:09.735260508Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:09.735926512Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:10.014946439Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:10.43057089Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:16:10.859439259Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:12.431207483Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:12.431803189Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:12.519987426Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:14.841852197Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:16.842314578Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:16.842776371Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:17.079225794Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:17.416535721Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:17.416826338Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:18.548464769Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:19.414506861Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:19.692402017Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:19.692926617Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:16:20.859507245Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:21.207230868Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:21.207598216Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:21.903436926Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:21.90360393Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:22.758203047Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:23.13497983Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:23.13523609Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:25.697689855Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:25.727046665Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:25.727280023Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:26.272401639Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:26.272693933Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:27.379571727Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:27.379901138Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:28.1723229Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:28.176025379Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:28.658257276Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:28.659028582Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:28.932435329Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:16:30.859415093Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:31.787251117Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:31.78752192Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:31.957893382Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:31.958303165Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:32.357153553Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:32.357529496Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:32.530232592Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:32.530771797Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:32.592146262Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:32.592591483Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:32.905585874Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:33.197899932Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:33.198324343Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:33.796060939Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:33.797144115Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:33.818186261Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:33.819307877Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:33.821558761Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:34.150851138Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:34.151126311Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:35.335705612Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:35.577262591Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:35.577716279Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:36.627114728Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:36.627491129Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:36.926171506Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:37.160753279Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:37.161176912Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:39.160959005Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:39.161415425Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:40.695786665Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:40.696049324Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:16:40.859457867Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:41.261040801Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:41.261361552Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:41.486737649Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:41.521054237Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:41.5213969Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:41.556903163Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:41.557206156Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:41.707831754Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:42.543541154Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:42.543995707Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:43.856468828Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:43.998090891Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:44.28270091Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:45.14059328Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:45.140998279Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:45.358102942Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:45.358909987Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:45.71384917Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:45.714537772Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:45.841796017Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:45.842413109Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:46.375927124Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:46.765463134Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:46.765863229Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:46.985128573Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:47.078836076Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:47.079223324Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:48.26068795Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:48.555088534Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:48.649954083Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:48.652279168Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:49.206644134Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:49.554839991Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:49.555113875Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:49.738912543Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:16:50.859499894Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:51.61483194Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:51.615538499Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:51.944450919Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:51.945010984Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:51.987882751Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:52.07558815Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:52.076052466Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:52.577389707Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:52.691793498Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:52.998981701Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:52.999511389Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:53.328006774Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:53.898287463Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:54.975157116Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:54.975376059Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:55.296016375Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:55.2964649Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:55.412915333Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:55.413076162Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:55.63824938Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:55.63883195Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:56.402967732Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:58.253440972Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:58.865229764Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:59.121409903Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:59.12180156Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:59.705366687Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:59.705647739Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:59.886508115Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:16:59.886684515Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:17:00.859452409Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:00.930649633Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:00.943200053Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:00.943636944Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:01.501649166Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:02.34510078Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:02.345262326Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:02.676371867Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:02.677063452Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:03.79181014Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:04.67926639Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:04.719959305Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:07.177935547Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:07.211968991Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:07.231491329Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:07.231754599Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:07.289770262Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:07.290602816Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:08.01238545Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:09.102842961Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:09.464647417Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:09.465086745Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:09.731410204Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:09.732440789Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:10.017382649Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:10.430803652Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:17:10.859436219Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:12.461537872Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:12.462129589Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:12.518887073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:14.841869053Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:16.72594667Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:16.726403707Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:17.080006076Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:17.435178291Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:17.435511178Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:18.191564137Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:19.407819139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:19.991173816Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:19.991716799Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:17:20.859438925Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:21.207643744Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:21.208038097Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:21.908267499Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:21.908448491Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:22.758728193Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:23.15410909Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:23.154361837Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:25.696847014Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:25.727690452Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:25.727979609Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:26.346081271Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:26.346344237Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:28.190403739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:28.192073572Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:28.723982943Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:28.724603159Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:28.933024448Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:17:30.859445222Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:31.787522086Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:31.787788441Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:32.275089809Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:32.27550219Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:32.349082369Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:32.349406395Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:32.515510018Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:32.516067107Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:32.592452384Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:32.592925985Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:32.905548834Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:33.196571938Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:33.196887891Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:33.795264656Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:33.796030747Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:33.81875266Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:33.819881825Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:33.8220858Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:34.150313724Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:34.150558592Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:35.311488461Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:35.576877935Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:35.577294875Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:36.605994068Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:36.606341205Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:36.919914026Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:37.1611355Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:37.161750284Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:37.302716459Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:37.302971301Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:39.135138144Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:39.135513985Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:40.687196105Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:40.687453913Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:17:40.85943452Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:41.304435315Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:41.304923028Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:41.48962139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:41.570797701Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:41.571118908Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:41.580535502Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:41.580875887Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:41.716584006Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:42.586351647Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:42.586934798Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:43.951252225Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:43.997728254Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:44.339880181Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:44.597272875Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:44.597657619Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:45.376766148Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:45.378530154Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:45.744597062Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:45.746059287Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:45.857019782Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:45.858321099Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:46.378531219Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:46.628090115Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:46.629019338Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:46.986007261Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:47.074150552Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:47.074479748Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:48.258101773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:48.555201537Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:48.646318637Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:48.647030859Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:49.20039991Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:49.557784871Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:49.558048191Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:49.732142033Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:17:50.859459037Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:51.629257951Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:51.630036276Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:51.903355181Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:51.904165711Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:51.986243003Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:52.068629829Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:52.069054954Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:52.576587498Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:52.692721059Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:52.998716952Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:52.999164692Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:53.328368119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:53.896335888Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:54.975168509Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:54.975361298Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:55.304999Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:55.305450175Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:55.412516158Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:55.412681601Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:55.650805266Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:55.651239356Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:56.402240939Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:58.254531042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:58.853234816Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:59.097404945Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:59.097762922Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:59.706493767Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:59.706776021Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:59.88674323Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:17:59.886889965Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:00.846658037Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:00.847234113Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:18:00.859492061Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:00.93076757Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:01.929893918Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:02.346929249Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:02.347109804Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:02.681202897Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:02.681933091Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:03.697384485Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:04.719910679Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:04.771592578Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:07.182468406Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:07.218992782Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:07.243296608Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:07.243545769Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:07.263920069Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:07.264697618Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:08.016685617Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:08.270275634Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:09.427595269Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:09.428440523Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:10.033734248Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:10.034317335Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:10.0655029Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:10.432026326Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:18:10.85943453Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:12.439450529Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:12.440041777Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:12.51717565Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:14.843865004Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:16.715170107Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:16.715633384Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:17.076829852Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:17.46243037Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:17.462767456Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:18.174295493Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:19.4083651Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:19.696479189Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:19.697017758Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:18:20.8594358Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:21.207775897Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:21.20810938Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:21.928407881Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:21.928580391Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:22.773368884Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:23.115995807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:23.116251318Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:25.69709788Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:25.726368234Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:25.726582708Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:26.300119655Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:26.300409337Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:27.384448624Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:27.384771236Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:28.198777845Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:28.199574642Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:28.838902196Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:28.839425951Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:28.94698188Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:18:30.859514202Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:31.786983668Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:31.78723219Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:32.281138257Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:32.281617902Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:32.354885599Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:32.355331561Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:32.475315547Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:32.475795346Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:32.592366941Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:32.592783319Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:32.905554496Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:33.21415604Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:33.214611769Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:33.795331339Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:33.796077636Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:33.816736737Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:33.817890858Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:33.820116225Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:34.150061544Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:34.15030985Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:35.310623033Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:35.587243006Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:35.587704381Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:36.613430512Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:36.613841673Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:36.947697773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:37.170538445Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:37.171135876Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:39.29395279Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:39.294433458Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:40.685784565Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:40.686023674Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:18:40.859489703Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:41.247300255Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:41.247590546Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:41.487103331Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:41.520124119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:41.520533478Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:41.55622393Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:41.556482391Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:41.709181455Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:42.550350078Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:42.550796219Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:43.854384616Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:43.99722197Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:44.27173624Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:45.138315465Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:45.13864188Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:45.358058275Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:45.358841202Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:45.713043703Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:45.713752081Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:45.913527534Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:45.914134582Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:46.376261908Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:46.599019275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:46.599377684Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:46.982111314Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:47.074242382Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:47.074561162Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:48.287904386Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:48.553578975Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:48.6742709Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:48.69880094Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:49.20082043Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:49.492742378Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:49.493001602Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:49.656386984Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:18:50.859447433Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:51.629995122Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:51.63071693Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:51.895916633Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:51.896540605Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:51.980417215Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:52.077639059Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:52.07806795Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:52.577937778Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:52.691637525Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:52.998894391Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:52.999513906Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:53.341427227Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:53.896206784Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:54.976905984Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:54.977151088Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:55.305917734Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:55.30638917Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:55.413026774Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:55.413180717Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:55.743002657Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:55.743392405Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:56.402907489Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:58.298030698Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:58.861244787Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:59.290797351Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:59.291122586Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:59.706856084Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:59.707202605Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:59.886914396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:18:59.887068104Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:19:00.859500324Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:00.930145123Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:01.059523641Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:01.060038697Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:01.617063226Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:02.37385305Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:02.374035504Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:02.692819192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:02.693443556Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:03.700595762Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:04.692008168Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:04.720350776Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:07.174679019Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:07.208835822Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:07.233421655Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:07.233727434Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:07.268813474Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:07.269557809Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:08.018964649Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:08.062022657Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:09.489029906Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:09.490192825Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:09.711022656Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:09.712108849Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:10.069168213Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:10.440497268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:19:10.859387091Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:12.44583608Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:12.446452071Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:12.597317758Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:14.841600635Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:17.071775152Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:17.290076356Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:17.290526262Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:17.43311268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:17.433398496Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:18.528237741Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:19.409204096Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:19.69404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:19.6945827Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:19:20.859412752Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:21.206767324Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:21.20713717Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:21.912293246Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:21.912480662Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:22.75820373Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:23.117112452Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:23.117383529Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:25.698555282Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:25.730732342Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:25.731066286Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:26.272086718Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:26.272376251Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:27.383926568Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:27.3842172Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:28.159608415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:28.160278552Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:28.779486261Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:28.779923469Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:28.935550287Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:19:30.8597434Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:31.786976026Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:31.787200859Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:32.310499054Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:32.310927424Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:32.457510717Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:32.457889577Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:32.533592193Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:32.534165049Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:32.572219341Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:32.572656918Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:32.905855785Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:33.2143485Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:33.214675973Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:33.795303568Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:33.796152331Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:33.81721079Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:33.818411744Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:33.820748498Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:34.16211844Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:34.162423071Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:35.312365852Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:35.577298145Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:35.577800018Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:36.6356486Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:36.636137351Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:36.95752631Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:37.161210382Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:37.161801507Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:39.120219774Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:39.120718771Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:40.700746415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:40.700988372Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:19:40.859436541Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:41.302902427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:41.303410552Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:41.49146373Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:41.546207254Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:41.546605637Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:41.58342432Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:41.600934809Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:41.802059563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:42.55512999Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:42.555541036Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:43.86014289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:43.997616869Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:44.338901781Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:45.127739637Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:45.128130173Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:45.358105182Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:45.3589193Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:45.728228773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:45.728961722Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:45.840518147Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:45.841360322Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:46.378203415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:46.605539519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:46.605910811Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:46.983371108Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:47.090342566Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:47.091112069Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:48.52903677Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:48.557572456Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:48.653932622Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:48.65478361Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:49.266813531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:49.530236253Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:49.530531004Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:49.657262594Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:19:50.859430097Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:51.703504445Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:51.704212449Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:51.919280927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:51.91996112Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:51.997690593Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:52.076348358Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:52.076778528Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:52.577235098Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:52.691929416Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:52.999010276Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:52.999457191Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:53.329890089Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:53.900956203Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:54.98341277Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:54.983594072Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:55.332095614Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:55.332539897Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:55.412301604Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:55.41242373Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:55.762028644Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:55.762399409Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:56.402573352Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:58.25498858Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:58.860243629Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:59.274860752Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:59.275151202Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:59.715500242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:59.715800116Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:59.889226013Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:19:59.889381297Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:20:00.859429556Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:00.930243388Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:01.036062801Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:01.036573919Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:02.685375635Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:02.722926229Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:03.298487135Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:03.299505973Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:03.679300977Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:03.705527534Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:04.676328017Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:04.722845162Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:07.262648152Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:07.313095135Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:07.352627825Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:07.353773984Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:07.36339772Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:07.36373325Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:08.036949716Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:09.719496747Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:09.720584118Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:09.913789239Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:09.91419257Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:10.010161114Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:10.470498446Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:10.500356355Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:20:10.859438417Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:12.442944711Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:12.44356801Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:12.528893811Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:14.841643568Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:16.841135574Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:16.841620318Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:17.170467359Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:17.479926063Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:17.480270719Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:18.234715275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:19.508253269Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:19.996710041Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:19.997218473Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:20:20.859424091Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:21.226303224Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:21.226636811Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:21.868034235Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:21.86821477Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:22.76475084Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:23.117514049Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:23.11787015Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:25.698062065Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:25.726596512Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:25.726873163Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:26.34677427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:26.347079309Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:27.311546527Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:27.311827236Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:28.160510426Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:28.161203901Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:28.698788568Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:28.699295209Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:28.955330914Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:20:30.859452504Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:31.787983808Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:31.788229355Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:32.102568047Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:32.103129657Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:32.476319099Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:32.476734321Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:32.496004053Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:32.496507362Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:32.594361116Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:32.594832443Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:32.905542989Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:33.226516131Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:33.227078651Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:33.795278887Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:33.796041719Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:33.816723107Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:33.817886283Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:33.820201307Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:34.150878506Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:34.151117365Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:35.312086663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:35.577684698Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:35.578154726Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:36.60761154Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:36.608094298Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:36.921268509Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:37.161321762Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:37.161901625Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:39.447058877Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:39.447525514Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:40.686570133Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:40.686833797Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:20:40.859426833Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:41.246843209Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:41.247329921Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:41.487310196Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:41.511209024Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:41.511794363Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:41.564450109Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:41.56480283Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:41.720598682Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:42.557868179Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:42.558298932Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:43.861088713Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:44.001061074Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:44.338963561Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:45.244462926Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:45.244803307Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:45.358120168Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:45.358931373Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:45.719232353Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:45.719963265Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:45.84107683Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:45.841716905Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:46.379866854Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:46.693934533Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:46.694282689Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:46.981900203Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:47.361532248Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:47.361890269Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:48.277096249Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:48.574982945Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:48.662367303Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:48.664979023Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:49.21423816Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:49.557380007Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:49.557625941Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:49.650743578Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:20:50.859508686Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:51.714141688Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:51.715585256Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:51.911226029Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:51.911899394Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:51.984130149Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:52.08145684Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:52.081924255Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:52.578887356Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:52.692945477Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:52.998758752Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:52.999178886Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:53.314820127Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:53.897313937Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:54.986211698Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:54.986469296Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:55.354536786Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:55.354986933Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:55.416800949Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:55.416955445Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:55.813734895Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:55.814306505Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:56.406307814Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:58.351554047Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:58.852205613Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:59.122255555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:59.122620898Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:59.711906517Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:59.712166521Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:59.888678827Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:20:59.888832841Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:21:00.859441699Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:00.931647994Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:00.980113925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:00.980599746Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:01.894398544Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:02.371279003Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:02.371508051Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:02.704159132Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:02.704861626Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:03.697928214Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:04.67734485Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:04.721538073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:07.21627111Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:07.262699805Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:07.309090459Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:07.309398437Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:07.32249921Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:07.323620299Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:08.013511541Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:08.935623195Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:09.482580655Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:09.483296792Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:09.710844006Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:09.711460728Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:10.030472637Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:10.435045722Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:21:10.859432082Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:12.442234274Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:12.442871633Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:12.519447118Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:14.842721128Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:16.842923979Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:16.843369912Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:17.078219096Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:17.430942803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:17.431334442Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:18.326237708Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:19.414027729Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:19.68405027Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:19.684544914Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:21:20.859425304Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:21.209208332Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:21.20958575Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:21.906325678Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:21.90650533Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:22.774292341Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:23.116953255Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:23.117191842Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:25.698045875Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:25.726395872Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:25.726625317Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:26.272033113Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:26.272312888Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:27.386276221Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:27.38655341Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:28.170013463Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:28.170607186Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:28.545780829Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:28.546324154Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:28.936796311Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:21:30.859496073Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:31.653358927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:31.653880196Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:31.787006614Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:31.787259707Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:31.964651611Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:31.965080027Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:32.479897978Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:32.480273706Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:32.592862018Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:32.593354127Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:32.90574925Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:33.224298567Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:33.224572823Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:33.795417419Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:33.796092681Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:33.815361118Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:33.816470777Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:33.818647131Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:34.150710006Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:34.150952108Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:35.311866394Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:35.577645447Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:35.578092998Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:36.643766489Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:36.644264351Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:36.920860879Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:37.160893519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:37.161368339Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:39.323769024Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:39.324152258Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:40.701300819Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:40.701618727Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:21:40.859456309Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:41.259397347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:41.259736615Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:41.487633351Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:41.534471674Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:41.534945592Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:41.557632185Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:41.557932826Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:41.717308295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:42.555813364Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:42.556278366Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:43.859914825Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:44.000810693Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:44.339080685Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:45.134829851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:45.135181382Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:45.358221553Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:45.359027393Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:45.716721937Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:45.717390956Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:45.863801383Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:45.864568177Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:46.376724302Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:46.615329031Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:46.615680618Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:46.980769987Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:47.219381401Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:47.219706751Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:48.261533368Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:48.558047788Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:48.641210041Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:48.642180379Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:49.21836165Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:49.558799092Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:49.559241898Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:49.645034307Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:21:50.859419091Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:51.710693779Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:51.711445427Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:51.891279611Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:51.89185097Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:51.98103456Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:52.086879282Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:52.08728856Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:52.577547744Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:52.691225196Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:52.998586309Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:52.999076851Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:53.300621524Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:53.897737099Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:54.971741418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:54.971967136Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:55.333146255Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:55.333638774Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:55.412368495Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:55.412509387Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:55.766269477Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:55.766698211Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:56.401896183Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:58.296257363Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:58.851915837Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:59.277677871Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:59.277997976Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:59.721346799Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:59.721684239Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:59.87080052Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:21:59.870949676Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:22:00.859428833Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:00.931155394Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:01.036770776Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:01.037242286Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:01.562541921Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:02.365641973Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:02.365811355Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:02.78074313Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:02.781538474Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:03.699589637Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:04.674300425Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:04.720156907Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:07.174484661Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:07.217218454Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:07.244832576Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:07.245144169Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:07.308693459Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:07.309744795Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:08.011896317Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:08.781181394Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:09.632741221Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:09.633286888Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:09.711275631Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:09.712126085Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:10.042965074Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:10.435408247Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:22:10.859426594Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:12.43881914Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:12.439441989Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:12.519160345Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:14.843649283Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:16.715182295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:16.71569614Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:17.077888391Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:17.442504418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:17.44291737Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:18.159228244Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:19.407854965Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:19.682078443Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:19.682606605Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:22:20.859447217Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:21.207454525Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:21.207819832Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:21.904215853Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:21.904378681Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:22.757708593Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:23.116857979Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:23.117112543Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:25.697368783Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:25.726308315Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:25.726540899Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:26.296931407Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:26.297236832Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:27.381581775Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:27.381916902Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:28.169456806Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:28.177636879Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:28.653787494Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:28.654322648Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:28.935191053Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:22:30.85950488Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:31.633037758Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:31.633619182Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:31.787427602Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:31.787702835Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:31.9608824Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:31.96130673Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:32.488145181Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:32.488528434Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:32.593603004Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:32.594203702Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:32.90561107Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:33.21400341Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:33.214316652Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:33.795275312Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:33.796018976Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:33.816486345Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:33.817646361Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:33.819901859Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:34.170892696Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:34.171103619Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:35.310470884Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:35.577185214Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:35.577646087Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:36.598915127Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:36.599310193Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:36.920107071Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:37.161297262Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:37.161869054Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:39.362253231Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:39.362656113Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:40.686098023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:40.686380274Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:22:40.85943708Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:41.144830869Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:41.145158871Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:41.487352707Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:41.520529657Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:41.520971594Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:41.55413202Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:41.554457538Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:41.721051999Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:42.216310873Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:42.216764428Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:43.859694777Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:43.998251141Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:44.346901354Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:44.566387482Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:44.566689101Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:45.358120335Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:45.358872603Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:45.724022411Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:45.724738584Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:45.836508887Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:45.837141988Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:46.377012794Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:46.637029161Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:46.637381711Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:46.985860241Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:47.230777Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:47.231079638Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:48.254658939Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:48.511018295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:48.511558679Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:48.55324462Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:49.201396615Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:49.555241808Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:49.555504553Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:49.660152333Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:22:50.859467895Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:51.739118119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:51.739792789Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:51.900420053Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:51.901100741Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:51.982396905Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:52.066556896Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:52.066993394Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:52.578061682Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:52.690826786Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:52.999497583Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:53.000100602Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:53.329144068Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:53.898470838Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:54.971086102Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:54.971303112Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:55.297310995Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:55.297829056Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:55.412353307Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:55.412480517Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:55.78531684Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:55.785695675Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:56.402052917Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:58.286794273Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:58.861728347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:59.298976791Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:59.299320066Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:59.716102959Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:59.716407755Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:59.872371504Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:22:59.872527374Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:23:00.859441639Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:00.934423653Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:01.098697419Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:01.099169827Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:01.661879439Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:02.358070553Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:02.358230717Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:02.674561796Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:02.675238023Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:03.698421673Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:04.676972222Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:04.720100905Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:07.168462799Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:07.200063426Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:07.220047489Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:07.220301757Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:07.292513538Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:07.293304984Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:08.024272364Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:08.170927394Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:08.928212052Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:08.928897896Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:09.703835934Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:09.704390078Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:10.015635595Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:10.434726863Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:23:10.859438183Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:12.432170249Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:12.43278977Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:12.51862715Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:14.847767133Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:16.674119325Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:16.674502221Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:16.714771677Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:16.715186258Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:17.077278571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:18.151161412Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:19.406424419Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:20.007424744Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:20.007913342Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:23:20.859431507Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:21.218384349Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:21.218752651Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:21.917051242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:21.917242726Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:22.75745553Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:23.116917162Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:23.117178375Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:25.697284703Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:25.726603667Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:25.726923957Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:26.345241269Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:26.345527707Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:27.38110667Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:27.381379568Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:28.168880396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:28.169426571Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:28.579078694Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:28.579569821Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:28.932197122Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:23:30.85942042Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:31.62054472Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:31.621112171Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:31.786863786Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:31.78712757Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:32.278595577Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:32.27909989Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:32.367744477Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:32.368199736Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:32.573994465Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:32.574686022Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:32.905541106Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:33.213595856Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:33.213926624Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:33.795239464Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:33.795988733Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:33.81571622Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:33.816910215Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:33.819313447Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:34.150581725Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:34.150868464Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:35.312640719Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:35.576843659Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:35.577290413Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:36.640468083Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:36.640999414Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:36.921407389Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:37.16588939Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:37.166339696Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:39.206119451Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:39.20649523Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:40.687085478Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:40.687321979Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:23:40.859499539Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:41.247075317Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:41.247368186Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:41.487616177Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:41.512591961Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:41.512993839Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:41.566121891Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:41.566516434Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:41.710560624Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:42.567812139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:42.568384799Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:43.857958992Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:43.997854931Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:44.338531869Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:45.12204514Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:45.122336445Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:45.35810777Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:45.358894462Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:45.730714157Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:45.731443739Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:45.831123317Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:45.832242732Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:46.374397516Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:46.624600342Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:46.625099295Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:46.986472932Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:47.079624695Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:47.080051696Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:48.256072136Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:48.521531638Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:48.522112487Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:48.553568925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:49.201072115Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:49.555020174Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:49.555237976Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:49.637103967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:23:50.86015738Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:51.739977901Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:51.741646457Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:51.901251332Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:51.901894143Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:52.001067672Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:52.143354636Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:52.143797457Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:52.578313295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:52.693412189Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:52.999148814Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:52.999579844Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:53.301015903Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:53.900095574Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:54.971309131Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:54.971514691Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:55.297779024Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:55.298206615Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:55.412756076Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:55.412901162Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:55.634479803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:55.634921633Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:56.405049803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:58.291349492Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:58.853519168Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:59.297380782Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:59.297803361Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:59.715734634Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:23:59.716018842Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:00.056106825Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:00.056272039Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:24:00.859436128Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:00.930394885Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:01.037749635Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:01.038248877Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:02.274080696Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:02.372543875Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:02.372836241Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:02.682404678Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:02.683166846Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:03.702198385Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:04.689749831Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:04.724247208Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:07.173288276Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:07.205423288Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:07.227508737Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:07.227872661Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:07.268516991Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:07.26923541Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:08.011156552Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:08.381420064Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:08.81324179Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:08.813624638Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:09.691817396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:09.692253602Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:10.033564271Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:10.440236005Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:24:10.862955783Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:12.435125051Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:12.435712034Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:12.598183967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:14.84082662Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:16.844876427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:16.845286074Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:17.445658443Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:17.446032309Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:18.447920289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:19.408412843Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:19.692181229Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:19.692717227Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:24:20.859496488Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:21.221683377Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:21.22200444Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:21.923235677Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:21.923428505Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:22.680516272Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:22.681204349Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:22.77352525Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:23.116151207Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:23.11640378Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:25.698360594Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:25.726289975Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:25.726555293Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:26.271951406Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:26.272264204Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:27.326865261Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:27.327151512Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:28.174768592Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:28.175385345Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:28.720380849Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:28.72095956Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:28.973773376Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:24:30.859443345Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:31.643113591Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:31.643632644Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:31.787361138Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:31.787620118Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:32.280863584Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:32.281291058Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:32.366980317Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:32.367380433Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:32.59763088Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:32.598133679Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:32.905560358Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:33.220576889Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:33.221270433Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:33.79525921Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:33.795990501Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:33.815928169Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:33.817077987Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:33.81922429Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:34.150315986Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:34.15055727Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:35.316138291Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:35.577152323Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:35.577645102Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:36.612912577Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:36.613291531Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:36.920725237Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:37.165455284Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:37.165861027Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:39.336905156Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:39.337381301Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:40.686001189Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:40.686276259Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:24:40.859507992Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:41.248491065Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:41.24882737Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:41.486610031Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:41.512620664Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:41.512953628Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:41.576119277Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:41.576387841Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:41.717202163Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:42.559579377Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:42.560011271Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:43.859047929Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:44.000231975Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:44.340975904Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:45.11715542Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:45.117496896Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:45.358142324Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:45.359088623Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:45.764763527Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:45.765406935Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:45.827714697Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:45.82830879Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:46.3750361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:46.64875972Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:46.649066423Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:46.986223408Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:47.060521559Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:47.060850376Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:48.26450742Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:48.523072228Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:48.523554285Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:48.560014197Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:49.200250227Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:49.503140748Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:49.503397389Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:49.643763562Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:24:50.859491407Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:51.710226518Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:51.710924354Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:51.917155638Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:51.917628229Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:51.981175931Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:52.090108474Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:52.090623991Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:52.577934639Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:52.691769454Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:53.002891302Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:53.003330107Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:53.316844361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:53.896850471Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:54.983970651Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:54.984204392Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:55.296835573Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:55.297421698Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:55.477636656Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:55.477847391Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:55.654972608Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:55.655408128Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:56.402610034Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:58.312553215Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:58.851631892Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:59.277925681Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:59.278232351Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:59.711286404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:59.711545867Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:59.870038194Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:24:59.870204496Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:25:00.859425768Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:00.930360084Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:01.031567881Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:01.03210033Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:01.574008471Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:02.638375677Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:02.646920726Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:02.976845266Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:02.978025984Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:03.699121171Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:04.675434956Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:04.720064277Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:07.179524126Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:07.218295726Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:07.239732725Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:07.239960005Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:07.316857383Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:07.31797507Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:08.019536903Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:08.742398167Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:09.642810072Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:09.643225677Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:09.700196727Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:09.700872951Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:10.019488521Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:10.430981465Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:25:10.85941963Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:12.460173213Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:12.460767995Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:12.516091236Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:14.847138412Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:16.845964516Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:16.846410827Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:17.078778572Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:17.417325308Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:17.417738001Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:18.393160194Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:19.407705902Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:19.708853115Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:19.709402449Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:25:20.859438147Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:21.223088637Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:21.223462233Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:21.891341866Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:21.891522375Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:22.772815081Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:23.116838763Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:23.117150226Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:25.700253792Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:25.744063522Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:25.744319566Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:26.272694007Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:26.273003531Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:27.381202192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:27.381437925Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:28.161938817Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:28.162516697Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:28.711970845Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:28.712419088Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:28.934293163Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:25:30.859427677Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:31.644589259Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:31.645174756Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:31.787017218Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:31.787255318Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:32.280134208Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:32.280506464Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:32.358249788Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:32.358634435Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:32.592933363Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:32.593380715Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:32.905519758Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:33.214252116Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:33.214595479Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:33.795222101Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:33.795975418Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:33.815696148Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:33.816822526Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:33.818973896Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:34.150562271Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:34.150952237Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:35.312047241Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:35.576976119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:35.577457941Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:36.631746127Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:36.632238261Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:36.924947955Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:37.160612094Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:37.161046272Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:39.24702611Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:39.247616656Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:40.68607728Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:40.686337929Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:25:40.859449093Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:41.259491223Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:41.25984114Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:41.414853785Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:41.4152468Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:41.487005707Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:41.55846771Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:41.558779058Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:41.728248171Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:42.553020047Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:42.553457773Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:43.865683124Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:43.998175071Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:44.272264109Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:45.12513479Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:45.125434849Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:45.358095656Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:45.358926726Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:45.713450972Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:45.714150619Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:45.842489398Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:45.843099957Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:46.380049037Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:46.780334225Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:46.780956095Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:46.980845603Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:47.07163145Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:47.071943379Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:48.255326395Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:48.553518148Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:48.641686539Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:48.642382312Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:49.201398568Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:49.493617485Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:49.493865161Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:49.641447015Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:25:50.859432205Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:51.716177703Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:51.716958052Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:51.982787019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:52.012151407Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:52.01328201Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:52.106876151Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:52.107892054Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:52.578734353Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:52.701547528Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:53.001515514Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:53.002535752Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:53.308227711Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:53.896361241Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:54.972699042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:54.97290657Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:55.296023578Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:55.29647169Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:55.4319621Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:55.432145829Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:55.744562198Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:55.745388302Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:56.402614677Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:58.285751432Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:58.858797178Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:59.287055576Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:59.287353708Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:59.713022565Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:59.713340848Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:59.890172024Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:25:59.89030958Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:26:00.85944713Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:00.931442068Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:01.030209681Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:01.030868255Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:01.535485702Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:02.352116461Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:02.352286899Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:02.683109672Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:02.683885798Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:03.698010788Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:04.682862817Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:04.720954068Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:07.186066111Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:07.21564116Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:07.234458669Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:07.234713441Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:07.328727628Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:07.329933944Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:08.012057231Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:08.554566549Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:09.394171683Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:09.394480237Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:09.691607981Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:09.692078043Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:10.011059706Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:10.430407536Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:26:10.859800417Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:12.432916581Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:12.433816142Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:12.517477872Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:14.842838877Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:16.726980578Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:16.727908763Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:17.077576184Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:17.428937224Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:17.429273666Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:18.155411684Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:19.417792471Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:19.99762616Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:19.998135504Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:26:20.85942992Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:21.208121947Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:21.20847695Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:21.900619382Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:21.900807855Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:22.773598664Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:23.116536148Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:23.1168074Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:25.702461811Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:25.755017762Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:25.755268912Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:26.344113962Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:26.34443267Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:27.381368506Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:27.381734297Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:28.173189128Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:28.173725061Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:28.728168771Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:28.728757877Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:28.962701613Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:26:30.859420608Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:31.791023256Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:31.791312282Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:32.281842215Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:32.282382986Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:32.494630041Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:32.495096899Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:32.530621789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:32.53119866Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:32.611738039Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:32.612216135Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:32.905539923Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:33.214031285Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:33.214379224Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:33.795337859Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:33.79607856Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:33.816477277Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:33.817613257Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:33.819879722Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:34.150277651Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:34.150531072Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:35.310734383Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:35.577214814Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:35.578112353Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:36.754211709Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:36.754613576Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:36.933934797Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:37.160807327Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:37.161218514Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:39.210122897Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:39.210616107Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:40.686571089Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:40.68681941Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:26:40.859441053Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:41.247903036Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:41.248207106Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:41.422137075Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:41.422509238Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:41.486968067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:41.605618044Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:41.605892557Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:41.718510969Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:42.552989008Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:42.553418668Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:43.955481137Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:43.998258126Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:44.337869587Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:45.126758082Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:45.127034303Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:45.358133041Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:45.358964484Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:45.714237183Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:45.714944012Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:45.854014683Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:45.8545701Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:46.377695708Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:46.765088047Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:46.765455436Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:46.983358547Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:47.060825751Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:47.061095116Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:48.283313739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:48.55419496Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:48.641426188Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:48.642319738Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:49.20020823Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:49.554471373Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:49.554767311Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:49.733306187Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:26:50.859441396Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:51.707354676Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:51.708023076Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:51.944956568Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:51.980100705Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:51.980631741Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:52.06789469Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:52.068244003Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:52.583776898Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:52.691547097Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:52.999321789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:52.999827625Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:53.315500476Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:53.896913019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:54.978927502Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:54.979131482Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:55.332110288Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:55.33260112Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:55.412610205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:55.412779783Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:55.765252497Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:55.765716906Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:56.411728184Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:58.287700355Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:58.854596262Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:59.147178922Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:59.147532445Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:59.72135967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:59.721628563Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:59.869350274Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:26:59.869515197Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:27:00.859414844Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:00.930273981Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:01.069194561Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:01.069627586Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:02.031385729Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:02.381278012Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:02.381462485Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:02.681480708Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:02.682205292Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:03.699594444Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:04.67749382Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:04.720487Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:07.178632785Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:07.213905512Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:07.232775074Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:07.23298565Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:07.261044443Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:07.261843084Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:08.017073658Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:08.298465496Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:09.382281961Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:09.38262822Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:09.685053546Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:09.685538426Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:10.013298901Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:10.496295456Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:27:10.863785554Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:12.440009709Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:12.440597279Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:12.596323364Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:14.846360894Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:16.733786897Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:16.734228007Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:17.075306087Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:17.414520249Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:17.414863406Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:18.142400134Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:19.40978662Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:19.689766332Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:19.690301098Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:27:20.859431464Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:21.334886308Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:21.335243903Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:21.903760981Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:21.90392393Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:22.758653139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:23.132643359Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:23.132949928Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:25.69839021Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:25.74625317Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:25.746508476Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:26.271748686Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:26.272009999Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:27.311265353Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:27.31151882Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:28.16968666Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:28.170270661Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:28.722127824Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:28.722629872Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:28.940458314Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:27:30.859426619Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:31.788424174Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:31.788696318Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:31.96310706Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:31.963452899Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:32.444805164Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:32.445165215Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:32.526296428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:32.526862178Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:32.592860062Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:32.593327323Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:32.905576527Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:33.214373699Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:33.214711204Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:33.795484364Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:33.796244872Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:33.820827018Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:33.82205997Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:33.82453181Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:34.150241855Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:34.150513101Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:35.310828967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:35.576891264Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:35.577388934Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:36.62362897Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:36.624143874Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:36.923875091Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:37.164062963Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:37.164999773Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:39.147477933Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:39.147863399Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:40.696932604Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:40.697167617Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:27:40.859458892Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:41.063764683Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:41.06408169Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:41.488155517Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:41.521153895Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:41.521537434Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:41.553984815Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:41.554242132Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:41.718013915Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:42.553316517Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:42.553774248Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:43.855826332Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:43.999227234Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:44.342769501Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:44.549215436Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:44.549514167Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:45.358077856Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:45.358851494Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:45.712783307Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:45.713425132Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:45.841304155Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:45.841942994Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:46.375200139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:46.595270407Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:46.595730869Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:46.982911115Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:47.077140929Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:47.077464127Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:48.260007822Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:48.552985726Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:48.669180375Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:48.669830079Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:49.205252075Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:49.556763217Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:49.556980711Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:49.639021312Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:27:50.859425077Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:51.709402347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:51.710134916Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:51.988696357Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:52.016040775Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:52.016565463Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:52.076973269Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:52.077363065Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:52.577462437Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:52.697348166Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:52.999232834Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:52.99974196Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:53.313907881Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:53.899323988Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:55.039013822Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:55.039236971Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:55.332547451Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:55.333010584Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:55.475815015Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:55.475997499Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:55.748912352Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:55.749344175Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:56.402080461Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:58.257059804Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:58.851625133Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:59.113392856Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:59.113783815Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:59.708498798Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:59.708799172Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:59.906050193Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:27:59.906177462Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:28:00.859430984Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:00.930155191Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:00.998714094Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:00.999170879Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:02.370236734Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:02.370435986Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:02.37712238Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:02.688462263Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:02.689176479Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:03.697995635Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:04.674598253Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:04.72448937Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:07.176596014Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:07.208161817Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:07.228071393Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:07.228341884Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:07.260184629Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:07.26096491Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:08.013338907Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:08.75058628Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:09.47448431Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:09.4749566Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:09.728124732Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:09.728943602Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:10.056348827Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:10.453044265Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:28:10.859453824Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:12.441959739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:12.442546598Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:12.518693186Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:14.841437457Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:16.841352082Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:16.841878239Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:17.074194513Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:17.43884878Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:17.439167601Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:18.155505851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:19.40662936Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:19.700602553Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:19.70112711Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:28:20.859416777Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:21.205986097Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:21.206313406Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:21.880912663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:21.881089736Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:22.774316427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:23.143941601Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:23.144196613Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:25.698198194Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:25.744080978Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:25.744322994Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:26.272048564Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:26.272311798Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:27.380096064Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:27.380411961Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:28.200843791Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:28.201445207Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:28.706575726Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:28.707109554Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:28.942593288Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:28:30.859447342Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:31.786890946Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:31.787103312Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:31.965360017Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:31.965765094Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:32.478638421Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:32.479051216Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:32.518199384Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:32.518754062Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:32.592423731Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:32.592850562Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:32.905569513Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:33.213619284Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:33.213998338Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:33.795399979Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:33.796168263Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:33.818615881Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:33.819776223Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:33.822010161Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:34.150529214Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:34.150795961Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:35.311858591Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:35.579361755Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:35.587370328Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:36.624979323Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:36.625468Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:36.921635906Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:37.171869797Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:37.17229403Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:39.168585684Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:39.169136481Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:40.695837519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:40.696151693Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:28:40.860915713Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:41.249529794Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:41.249888533Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:41.487202404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:41.520026174Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:41.520446412Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:41.560773116Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:41.561129751Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:41.717331478Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:42.556331949Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:42.55710067Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:43.863483329Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:43.997855942Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:44.275016062Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:45.125731691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:45.1260743Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:45.358111454Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:45.358953393Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:45.714302011Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:45.715412822Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:45.842388924Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:45.843036554Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:46.375515176Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:46.605383774Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:46.605736879Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:46.98562285Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:47.059076435Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:47.059338343Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:48.259826027Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:48.553355247Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:48.640249607Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:48.640889721Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:49.201366512Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:49.555225296Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:49.555463283Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:49.643893094Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:28:50.859432118Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:51.710275873Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:51.711440142Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:51.97627517Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:51.97681085Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:51.982338068Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:52.078998458Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:52.07938475Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:52.576855328Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:52.691199794Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:52.999285084Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:52.99976814Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:53.300628852Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:53.8982438Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:54.981026353Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:54.981255016Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:55.331986489Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:55.332468525Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:55.412548225Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:55.412704274Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:55.772959723Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:55.773333748Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:56.402209007Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:58.277186367Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:58.854546438Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:59.276208385Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:59.276534221Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:59.716025011Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:59.716281045Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:59.891729664Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:28:59.891950709Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:29:00.859428684Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:00.930785967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:01.081866948Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:01.082332827Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:01.699685078Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:02.367356242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:02.367514381Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:02.678039711Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:02.678743555Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:03.698756348Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:04.67979109Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:04.720048581Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:07.179884706Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:07.210336096Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:07.235270926Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:07.2355259Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:07.267185723Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:07.267877035Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:08.010865386Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:08.233796261Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:09.379601914Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:09.379928902Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:09.673111934Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:09.673564601Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:10.011831225Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:10.43109663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:29:10.859448248Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:12.43423842Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:12.434870315Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:12.597484447Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:14.844434669Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:16.84101364Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:16.841462679Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:17.073859153Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:17.408853791Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:17.409112025Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:18.362097269Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:19.405300333Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:19.98930977Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:19.989815302Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:29:20.861758547Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:21.224787966Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:21.225112549Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:21.919702435Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:21.919866882Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:22.757581815Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:23.116648538Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:23.116966759Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:25.698058405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:25.744474031Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:25.744727842Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:26.343889578Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:26.344192102Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:27.380561732Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:27.380875157Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:28.16077397Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:28.161357161Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:28.563197035Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:28.563637709Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:28.935111558Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:29:30.859423736Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:31.787123205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:31.787414912Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:32.297513927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:32.297945785Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:32.454475292Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:32.454882178Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:32.519687258Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:32.520222765Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:32.592397211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:32.592804536Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:32.905577178Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:33.214848364Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:33.215142662Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:33.795316661Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:33.796064754Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:33.81590665Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:33.817050209Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:33.819185174Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:34.150328227Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:34.150558387Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:35.312593032Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:35.586769197Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:35.587367178Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:36.639328504Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:36.639916895Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:36.923587931Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:37.16981483Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:37.170276681Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:39.228155165Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:39.228567479Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:40.701181408Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:40.701423864Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:29:40.859452754Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:41.243155811Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:41.24348408Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:41.437360736Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:41.437765573Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:41.490084266Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:41.565772179Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:41.566079995Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:41.737983497Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:42.357399969Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:42.357841108Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:43.855378344Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:44.000637963Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:44.338744807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:45.135698647Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:45.135997357Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:45.358217381Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:45.359004291Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:45.719691451Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:45.720367351Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:45.828849434Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:45.82946164Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:46.379742311Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:46.60252118Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:46.602883478Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:46.980837699Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:47.059560608Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:47.059877575Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:48.257383103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:48.552654685Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:48.655712474Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:48.656301709Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:49.201291997Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:49.557584732Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:49.557896994Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:49.645890095Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:29:50.859440214Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:51.709426104Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:51.710133678Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:51.977483065Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:51.978000357Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:51.981182929Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:52.08659281Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:52.08698383Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:52.576869933Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:52.691686883Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:52.998956231Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:52.999339192Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:53.323695043Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:53.900545739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:54.977522171Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:54.978023275Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:55.359246603Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:55.36006586Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:55.485120684Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:55.485432928Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:55.949866701Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:55.951092072Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:56.405816837Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:58.259560874Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:58.851612221Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:59.288147785Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:59.288474968Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:59.719931735Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:59.720268699Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:59.885710908Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:29:59.885865398Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:30:00.859422324Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:00.930816691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:01.012353282Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:01.012856445Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:01.591416073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:02.759492107Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:02.770898867Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:04.098551275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:04.73419667Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:04.788998394Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:04.856418969Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:04.857382117Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:07.175376987Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:07.206111494Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:07.225631386Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:07.225886484Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:07.261748869Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:07.262500942Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:08.030180498Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:09.583794131Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:09.584074888Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:09.69106391Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:09.691555185Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:10.013839576Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:10.357447839Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:10.433867575Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:30:10.859423617Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:12.525908512Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:12.526744268Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:12.695842755Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:14.846793914Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:16.941030887Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:16.941624617Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:17.076337799Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:17.133480462Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:17.133817386Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:18.574576166Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:19.407409481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:19.686351585Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:19.687236014Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:30:20.859424618Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:21.218405879Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:21.218814398Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:21.954814008Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:21.954993911Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:22.761146455Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:23.117922724Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:23.118149535Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:25.698278598Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:25.744527068Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:25.744824217Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:26.272378466Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:26.272658408Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:27.31102474Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:27.31130218Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:28.163958563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:28.164560531Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:28.546057847Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:28.546498848Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:28.958460219Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:30:30.859437613Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:31.787237721Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:31.787494455Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:32.422140965Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:32.42253784Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:32.463979966Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:32.464333913Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:32.545486168Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:32.546042017Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:32.592930233Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:32.593354235Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:32.905609797Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:33.213658638Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:33.213999528Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:33.79545242Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:33.796274901Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:33.816393973Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:33.818243146Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:33.821465977Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:35.311044954Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:35.577029967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:35.577461933Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:35.647734062Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:35.647997785Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:36.626455945Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:36.627000136Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:36.92307828Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:37.161340603Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:37.16180548Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:39.540350377Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:39.540872642Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:40.70047053Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:40.700745693Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:30:40.859514869Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:41.147145959Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:41.147488042Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:41.488178162Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:41.515623511Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:41.515966712Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:41.59001048Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:41.59027613Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:41.723774916Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:42.5480815Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:42.548516285Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:43.872081787Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:44.002376075Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:44.337692215Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:45.180968681Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:45.181411754Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:45.358136606Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:45.359068352Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:45.718300549Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:45.718983829Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:45.842122294Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:45.843122101Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:46.375857644Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:46.688019234Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:46.688389387Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:46.982703794Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:47.084757428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:47.085081344Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:48.261298061Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:48.555978509Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:48.653588272Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:48.654308361Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:49.228166114Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:49.537828917Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:49.539606726Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:49.646438052Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:30:50.859435578Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:51.714506583Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:51.716085055Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:51.925679531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:51.926177247Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:51.981632889Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:52.085954709Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:52.086475404Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:52.584282651Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:52.691789592Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:52.998914313Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:52.999333446Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:53.330377566Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:53.900458478Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:55.0482921Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:55.048521974Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:55.390640303Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:55.391098516Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:55.480188647Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:55.480368575Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:55.691116103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:55.69170798Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:56.402449044Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:58.260860999Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:58.856515885Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:59.106443664Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:59.106770075Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:59.714379866Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:59.714704447Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:59.893042561Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:30:59.893204359Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:31:00.859428768Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:00.900519632Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:00.901025571Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:00.931249029Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:01.598101798Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:02.360684021Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:02.360846366Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:02.67069165Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:02.671316528Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:03.702659309Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:04.723808405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:04.769921953Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:07.210161371Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:07.241683178Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:07.263215806Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:07.263443218Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:07.31334726Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:07.314748702Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:08.012263685Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:09.422208313Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:09.422608819Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:09.70941967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:09.710030883Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:10.0117501Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:10.432962075Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:31:10.86172243Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:11.421540556Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:12.441355757Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:12.441994651Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:12.517630397Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:14.848333815Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:16.736544193Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:16.73711991Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:17.077862645Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:17.419078616Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:17.419356118Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:18.404168132Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:19.417599303Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:19.69817055Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:19.698862762Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:31:20.859442997Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:21.2242748Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:21.224627979Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:21.929429666Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:21.929610221Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:22.772820527Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:23.116859434Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:23.117130814Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:25.697863195Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:25.744152865Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:25.744420308Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:26.294391085Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:26.294695127Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:27.378583491Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:27.378848093Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:28.16100931Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:28.161572551Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:28.538959492Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:28.539581429Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:28.931713738Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:31:30.859582474Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:31.644231011Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:31.644720518Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:31.787149788Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:31.787388152Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:32.28097942Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:32.28146417Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:32.36552221Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:32.365892716Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:32.592854773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:32.593283962Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:32.905470966Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:33.213802876Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:33.214097425Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:33.795238509Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:33.796061403Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:33.817870539Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:33.819030124Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:33.821208679Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:34.150425955Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:34.151884733Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:35.311881144Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:35.576923476Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:35.577403256Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:36.739389152Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:36.740407144Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:36.926819944Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:37.16263648Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:37.163254917Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:39.261321124Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:39.263092946Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:40.686483704Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:40.686889935Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:31:40.859432407Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:41.267029371Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:41.267353419Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:41.488753792Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:41.543510613Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:41.543934595Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:41.583128722Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:41.583636049Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:41.708690049Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:42.550901466Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:42.551361793Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:43.859997004Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:43.999332902Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:44.338053502Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:45.140926262Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:45.141222104Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:45.358223635Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:45.359269025Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:45.730414119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:45.731354182Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:45.844523999Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:45.845193892Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:46.380050598Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:46.602747983Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:46.603083376Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:46.984869207Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:47.076439797Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:47.07691864Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:48.311609658Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:48.520097133Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:48.520571423Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:48.553209023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:49.200974703Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:49.493031983Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:49.493254732Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:49.646837632Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:31:50.85942987Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:51.708649992Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:51.709368948Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:51.989605526Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:51.993980499Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:51.994480287Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:52.07476871Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:52.075221934Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:52.577261157Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:52.694483626Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:52.998773701Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:52.999187085Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:53.338045005Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:53.898077566Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:54.9751837Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:54.975414127Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:55.295853801Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:55.296303157Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:55.481627341Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:55.481887566Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:55.661243783Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:55.661686855Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:56.402486342Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:58.269065432Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:58.859316637Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:59.276095906Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:59.27641134Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:59.713133529Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:59.713427378Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:59.871456996Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:31:59.871624021Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:32:00.859432931Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:00.931340744Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:00.991507927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:00.991981302Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:02.208194536Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:02.378575558Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:02.378789336Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:02.689305572Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:02.690091459Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:03.698421415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:04.675584582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:04.720515206Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:07.179965045Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:07.211847371Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:07.237683441Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:07.237971395Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:07.267982686Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:07.268698716Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:08.015160965Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:08.673379882Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:08.795784499Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:08.796105121Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:09.726810943Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:09.727428426Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:10.014548002Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:10.431031428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:32:10.859439847Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:12.431096797Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:12.431636625Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:12.595656935Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:14.845866093Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:16.620535384Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:16.621067748Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:16.854899371Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:16.855344165Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:17.078048748Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:18.114041851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:19.412123252Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:19.994059864Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:19.994582906Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:32:20.859421792Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:21.208332214Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:21.208684726Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:21.923869663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:21.924010199Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:22.77285243Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:23.116431289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:23.116721348Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:25.698068747Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:25.74591579Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:25.746108188Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:26.348505042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:26.348781583Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:27.3849416Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:27.385215878Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:28.160160586Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:28.160761159Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:28.702352877Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:28.703012916Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:28.967572371Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:32:30.859434706Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:31.651258616Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:31.65207396Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:31.787434206Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:31.787800693Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:32.278909241Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:32.279408374Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:32.377186198Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:32.377640063Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:32.564631758Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:32.565119286Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:32.905609729Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:33.226304414Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:33.226610596Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:33.7954318Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:33.796330009Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:33.815868303Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:33.817020437Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:33.819222388Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:34.150606767Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:34.150858716Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:35.311388616Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:35.576916892Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:35.577382789Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:36.628717631Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:36.629210724Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:36.937011838Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:37.160657456Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:37.161051753Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:39.255883502Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:39.256239152Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:40.719657531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:40.720061656Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:32:40.860227121Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:41.278620054Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:41.278912665Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:41.487097462Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:41.513690918Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:41.514044318Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:41.559206378Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:41.559510117Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:41.719355045Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:42.220586598Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:42.221251172Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:43.862761452Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:44.000659043Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:44.342774918Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:44.551273338Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:44.551582803Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:45.358096776Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:45.35889889Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:45.714360494Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:45.715052325Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:45.830330098Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:45.830959782Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:46.376253857Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:46.637073092Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:46.63750939Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:46.986381928Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:47.083953789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:47.084429041Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:48.257360232Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:48.51095722Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:48.511420051Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:48.553936975Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:49.200846442Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:49.504224745Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:49.504433369Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:49.638272076Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:32:50.859430602Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:51.706330689Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:51.707052023Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:51.931390571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:51.932106537Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:51.988501275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:52.099396783Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:52.099905165Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:52.578123872Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:52.693190019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:52.998938279Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:52.999384056Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:53.329335345Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:53.897081003Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:54.991861422Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:54.992095494Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:55.295892019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:55.296362046Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:55.413131543Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:55.413296835Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:55.756594462Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:55.757049843Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:56.41163528Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:58.260876684Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:58.851737437Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:59.286799377Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:59.287090707Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:59.712425009Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:59.712735165Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:59.872898988Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:32:59.873060785Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:33:00.859441172Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:00.931277607Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:01.083424119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:01.083916635Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:01.906153576Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:02.347270773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:02.34746017Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:02.771467139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:02.772138567Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:03.699836094Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:04.676765119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:04.720267822Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:07.172150755Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:07.233386828Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:07.253023518Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:07.253231568Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:07.274124006Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:07.27488029Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:08.020951975Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:08.046715906Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:09.56059529Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:09.561017754Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:09.702982436Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:09.703513612Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:10.033061172Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:10.430937626Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:33:10.859498595Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:12.466505867Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:12.467111788Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:12.599130649Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:14.841017427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:16.841798085Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:16.842247565Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:17.071268841Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:17.404557869Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:17.404965587Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:18.238237868Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:19.409828376Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:19.683290273Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:19.683851655Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:33:20.859417673Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:21.210016091Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:21.210377378Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:21.899309098Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:21.899454975Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:22.758640566Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:23.116193991Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:23.116502664Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:25.702886424Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:25.744005719Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:25.74423418Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:26.30074627Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:26.301051488Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:27.378652396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:27.378948483Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:28.159512425Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:28.160359835Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:28.726372735Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:28.727135621Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:28.94057328Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:33:30.859492851Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:31.628000071Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:31.628525635Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:31.769327268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:31.769555272Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:31.964411816Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:31.964846426Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:32.343244943Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:32.343612799Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:32.593031739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:32.593498575Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:32.905567602Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:33.214969124Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:33.215261291Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:33.795266106Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:33.796011924Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:33.81658659Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:33.818194036Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:33.820437645Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:34.150484078Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:34.150728002Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:35.312796042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:35.576915672Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:35.577402342Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:36.635623088Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:36.636126389Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:36.922432914Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:37.167867455Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:37.168291023Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:39.169793284Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:39.170133345Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:40.713651633Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:40.713997728Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:33:40.859380285Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:41.274326785Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:41.274683488Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:41.488167721Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:41.521354168Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:41.521741744Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:41.573325349Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:41.573731766Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:41.708231948Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:42.547682526Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:42.548130399Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:43.862723536Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:43.998899187Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:44.276334971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:45.116498957Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:45.116832776Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:45.358102299Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:45.358897665Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:45.76852888Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:45.769202223Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:45.879127086Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:45.879750851Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:46.374487718Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:46.675317697Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:46.675844048Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:46.984882224Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:47.225884481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:47.226201962Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:48.255069701Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:48.511240577Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:48.511723806Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:48.592342008Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:49.21225734Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:49.557764714Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:49.55802215Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:49.649840363Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:33:50.859437877Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:51.737967029Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:51.738631588Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:51.925176678Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:51.925638767Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:51.98199791Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:52.071751216Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:52.07214937Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:52.577167197Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:52.693809425Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:52.998848538Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:52.999282876Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:53.343331841Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:53.896585843Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:54.974890447Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:54.975072623Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:55.333639818Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:55.334150013Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:55.413282031Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:55.41341926Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:55.798295244Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:55.79874698Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:56.401917518Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:58.269567548Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:58.854879734Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:59.325919219Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:59.326755813Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:59.7893771Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:33:59.789883401Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:00.043785801Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:00.044101598Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:34:00.859432961Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:00.899293974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:00.899842865Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:00.930744608Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:01.581872716Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:02.35918045Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:02.359357401Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:02.719845803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:02.720770632Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:03.698252325Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:04.677252686Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:04.721965137Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:07.18636299Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:07.219082576Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:07.238726379Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:07.238964893Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:07.267138789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:07.267894405Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:08.010584515Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:08.492171255Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:09.469063626Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:09.469478926Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:09.692686217Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:09.693218946Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:10.046205942Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:10.432190766Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:34:10.859466848Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:12.439576392Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:12.44014909Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:12.517154278Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:14.844314314Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:16.841598057Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:16.842063195Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:17.075852708Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:17.441653798Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:17.441997186Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:18.500098789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:19.405964242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:19.687453179Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:19.688039312Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:34:20.85942874Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:21.210349106Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:21.210764519Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:21.944421358Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:21.944605874Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:22.761491222Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:23.115788383Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:23.116025411Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:25.697215674Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:25.743908519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:25.744128387Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:26.27299489Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:26.27333051Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:28.165867492Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:28.166429064Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:28.745517302Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:28.746239817Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:28.942621218Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:34:30.859499743Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:31.643764881Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:31.644294061Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:31.769118711Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:31.769364221Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:31.962135429Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:31.962508569Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:32.450143834Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:32.450528016Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:32.602891953Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:32.60337733Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:32.905579437Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:33.214427529Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:33.214775164Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:33.795393536Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:33.796196038Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:33.817042639Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:33.818194545Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:33.820386405Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:34.150350974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:34.150618024Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:35.311632772Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:35.577711847Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:35.578322451Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:36.612852732Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:36.613265264Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:36.920743247Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:37.160862489Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:37.16124044Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:37.302625532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:37.302928578Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:39.292400114Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:39.293174378Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:40.685489029Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:40.68572798Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:34:40.859430008Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:41.249243951Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:41.249559103Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:41.486738812Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:41.509249039Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:41.509615596Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:41.556225349Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:41.556571342Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:41.719338067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:42.574919132Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:42.575355876Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:43.869143114Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:43.997755132Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:44.337992443Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:45.130527488Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:45.130847193Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:45.358125304Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:45.358954885Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:45.7341208Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:45.734763298Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:45.923626348Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:45.92426783Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:46.37750219Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:46.604318295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:46.604706106Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:46.986651335Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:47.215102389Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:47.215390175Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:48.255601456Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:48.553003807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:48.651447555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:48.655174777Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:49.203142186Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:49.556790625Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:49.556999675Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:49.639314641Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:34:50.859430695Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:51.752200149Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:51.752934017Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:51.892846177Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:51.893372788Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:51.942159284Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:52.06840919Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:52.068789584Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:52.577688067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:52.69094762Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:52.998820114Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:52.999267953Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:53.332284878Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:53.899033928Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:54.976831959Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:54.977061139Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:55.338179839Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:55.338628528Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:55.413050563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:55.413213522Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:55.756433897Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:55.757159591Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:56.403883806Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:58.263614417Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:58.854487713Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:59.312782291Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:59.313069001Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:59.728947609Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:59.72936206Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:59.885945623Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:34:59.886094231Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:35:00.859753738Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:00.880505431Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:00.881147906Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:00.930693443Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:02.37996388Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:02.698235333Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:02.700502538Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:03.140440041Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:03.140571804Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:03.797035882Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:04.676532021Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:04.720404558Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:07.173347707Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:07.202974851Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:07.223373129Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:07.223616044Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:07.261439609Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:07.262232522Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:08.012132418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:08.287956175Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:09.467914634Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:09.468308906Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:09.855847414Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:09.85643445Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:10.026888278Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:10.431543535Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:35:10.859516185Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:12.442394594Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:12.443089394Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:12.600183846Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:14.841181693Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:16.724284975Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:16.724795398Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:17.076676752Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:17.252072678Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:17.252382468Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:18.159450101Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:19.409127255Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:19.990925201Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:19.991453025Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:35:20.859412469Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:21.234709067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:21.235157953Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:21.983429103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:21.983607352Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:22.772871646Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:23.116487726Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:23.116772893Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:25.698272569Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:25.744518364Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:25.744787561Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:26.342711647Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:26.342990682Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:27.38182345Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:27.382122028Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:28.160079266Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:28.160926009Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:28.722897013Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:28.723336048Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:28.934366249Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:35:30.859519544Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:31.650610655Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:31.651201765Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:31.769229439Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:31.769477563Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:32.273640856Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:32.274042801Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:32.445830179Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:32.446154863Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:32.592527872Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:32.593220273Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:32.905429179Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:33.214399907Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:33.215065781Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:33.795309213Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:33.796132985Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:33.815768539Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:33.816901173Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:33.81912733Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:34.150565687Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:34.150809439Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:35.314278132Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:35.577037761Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:35.577511826Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:36.665994539Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:36.666427071Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:36.950299554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:37.168283349Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:37.168723753Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:39.107698671Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:39.108152216Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:40.696637404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:40.697045258Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:35:40.861692778Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:41.26399361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:41.264328675Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:41.486807533Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:41.520485308Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:41.520852343Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:41.566241224Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:41.566573473Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:41.749856386Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:42.540198869Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:42.540603854Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:43.860657573Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:43.997434934Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:44.338074173Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:45.124455524Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:45.124852991Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:45.358077834Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:45.358845654Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:45.720656925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:45.721288651Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:45.925103942Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:45.925753588Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:46.37446376Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:46.764700181Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:46.765142322Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:46.983180307Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:47.218298364Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:47.21865269Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:48.257247196Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:48.554923023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:48.687053956Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:48.688204278Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:49.200653207Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:49.557687639Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:49.557948182Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:49.740584053Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:35:50.859431331Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:51.716881495Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:51.717641565Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:51.950440364Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:51.964158076Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:52.086076302Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:52.223688746Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:52.224168473Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:52.583265437Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:52.692012755Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:53.020624042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:53.021087953Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:53.350550076Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:53.897928562Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:54.979627523Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:54.979850687Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:55.33394635Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:55.334372251Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:55.43149168Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:55.431639355Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:55.762553931Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:55.762979189Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:56.404291603Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:58.273199844Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:58.856293489Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:59.274778608Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:59.275103168Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:59.712052128Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:59.712330543Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:59.888767322Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:35:59.888933865Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:36:00.860572774Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:00.942246275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:01.059956147Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:01.061125962Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:02.193137105Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:02.370106824Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:02.370412596Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:02.750566068Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:02.752014411Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:03.699198211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:04.681073792Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:04.720710782Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:07.179942392Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:07.209102672Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:07.229183841Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:07.22944461Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:07.269371519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:07.270374949Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:08.011848027Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:09.591152307Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:09.5934438Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:09.638265653Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:09.706446916Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:09.706924124Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:10.043768227Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:10.434769169Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:36:10.85951812Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:12.440726039Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:12.441339799Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:12.522371907Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:14.961141192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:16.715175364Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:16.715603482Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:17.079755596Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:17.435692243Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:17.436314954Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:18.292981942Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:19.407450267Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:19.700995281Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:19.701517417Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:36:20.85943826Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:21.219132421Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:21.21959024Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:21.899469214Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:21.90013652Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:22.759568242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:23.128863357Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:23.129141318Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:25.696871739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:25.743836651Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:25.744099441Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:26.271528742Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:26.271851879Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:27.379514459Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:27.379770614Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:28.160223835Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:28.160885429Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:28.532628529Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:28.533087751Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:29.014426088Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:36:30.859428793Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:31.772289593Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:31.772543589Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:32.278412442Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:32.2788055Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:32.476452345Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:32.47682704Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:32.536762994Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:32.537333987Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:32.592750102Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:32.593161052Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:32.905756542Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:33.213779042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:33.214166216Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:33.795341458Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:33.796078423Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:33.815273948Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:33.816388572Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:33.818549128Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:34.132550588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:34.132812203Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:35.309605296Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:35.577312738Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:35.577801666Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:36.629358467Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:36.629796862Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:36.946867715Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:37.161019634Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:37.161471044Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:39.312821008Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:39.313220756Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:40.687531257Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:40.687873578Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:36:40.859453998Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:41.251008982Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:41.251401225Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:41.491654386Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:41.517132353Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:41.517473093Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:41.557073027Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:41.557370475Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:41.722151714Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:42.543323662Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:42.543791032Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:43.858686502Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:43.997136342Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:44.338201892Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:45.136205158Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:45.136485678Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:45.358245952Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:45.358995639Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:45.72842162Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:45.729084542Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:45.925543531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:45.926186589Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:46.374872008Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:46.60784189Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:46.608175338Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:46.983299049Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:47.07750857Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:47.077828262Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:48.254149966Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:48.55593602Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:48.649007287Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:48.649591471Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:49.200336476Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:49.556717755Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:49.556970424Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:49.685256124Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:36:50.859467615Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:51.70032796Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:51.701075734Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:51.91320963Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:51.913871094Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:51.983167418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:52.06630952Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:52.066755206Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:52.577521841Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:52.691735966Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:52.998790274Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:52.999242188Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:53.328520165Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:53.897204364Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:54.974326343Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:54.974544135Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:55.33302683Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:55.33344666Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:55.413925377Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:55.414049161Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:55.772619258Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:55.773054741Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:56.404229354Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:58.307423066Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:58.851597773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:59.081411204Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:59.081769598Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:59.712449922Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:59.712765978Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:59.886216779Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:36:59.886374249Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:37:00.859394561Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:00.930175585Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:01.030255875Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:01.030739107Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:01.492122111Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:02.367368758Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:02.367533636Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:02.68720492Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:02.687992408Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:03.698631692Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:04.686937953Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:04.720121713Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:07.18396213Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:07.218961053Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:07.238940839Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:07.239171121Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:07.266291997Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:07.267030523Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:08.012913634Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:08.484902315Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:09.394972596Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:09.395394657Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:09.709335765Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:09.710143118Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:10.037055662Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:10.430782322Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:37:10.859499453Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:12.455607962Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:12.456417438Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:12.595510764Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:14.840195701Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:16.84175918Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:16.842191461Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:17.070808619Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:17.433986279Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:17.434312886Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:18.158604056Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:19.407563985Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:19.698375237Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:19.698878293Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:37:20.859502342Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:21.20765782Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:21.207998559Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:21.941767277Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:21.941923192Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:22.77724499Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:23.11673955Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:23.117011138Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:25.698394602Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:25.753697648Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:25.753957047Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:26.272562307Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:26.272869388Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:27.326422254Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:27.326703946Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:28.175079724Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:28.175721501Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:28.54848162Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:28.5490491Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:28.932885365Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:37:30.859431334Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:31.773265765Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:31.773495541Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:32.279990533Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:32.280321277Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:32.47070266Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:32.471136753Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:32.501432371Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:32.501962744Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:32.609040838Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:32.609599597Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:32.905527079Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:33.213680143Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:33.214012962Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:33.795463633Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:33.79620995Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:33.81610384Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:33.817216618Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:33.819392645Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:34.133299572Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:34.133596958Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:35.311215806Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:35.576884941Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:35.577344993Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:36.692996641Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:36.69342951Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:36.921310282Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:37.16239847Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:37.162832052Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:39.11424598Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:39.114654059Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:40.685760131Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:40.686007258Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:37:40.859425168Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:41.049737342Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:41.050066137Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:41.487712865Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:41.505291721Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:41.505617801Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:41.556344088Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:41.556610968Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:41.719808355Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:42.565984582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:42.566389688Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:43.859271371Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:43.998639541Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:44.275274266Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:44.550745108Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:44.551025576Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:45.358090914Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:45.358837339Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:45.728919939Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:45.729803377Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:45.934865173Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:45.93551971Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:46.374239734Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:46.607443086Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:46.607785606Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:46.986413932Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:47.074586237Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:47.074931042Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:48.260609415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:48.553151977Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:48.64760376Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:48.648434424Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:49.202263209Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:49.493442342Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:49.493765633Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:49.656585975Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:37:50.859465748Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:51.709345327Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:51.710096705Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:51.892625507Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:51.893258044Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:51.989789418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:52.086915316Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:52.087429518Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:52.576776585Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:52.693118667Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:52.998820138Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:52.999272818Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:53.300653261Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:53.896452554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:54.970800874Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:54.971007006Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:55.296746054Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:55.297229509Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:55.413817925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:55.41398303Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:55.716847161Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:55.717399247Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:56.420405425Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:58.280852024Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:58.853991687Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:59.098616568Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:59.098934396Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:59.710741826Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:59.711048272Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:59.886879161Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:37:59.887013588Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:38:00.859510348Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:00.930107305Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:01.102355273Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:01.102861652Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:01.876591769Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:02.371112293Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:02.371273666Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:02.773187097Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:02.774807952Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:03.700364632Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:04.685245199Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:04.720833279Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:07.170855945Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:07.200339391Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:07.218757747Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:07.218984314Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:07.2693449Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:07.270146083Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:08.017653745Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:09.146310441Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:09.455141949Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:09.455775814Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:09.717962317Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:09.718532441Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:10.026215334Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:10.431276765Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:38:10.859429329Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:12.438912594Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:12.439510868Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:12.595984438Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:14.844672973Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:16.841544517Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:16.842029949Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:17.074697189Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:17.434897254Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:17.435233926Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:18.171262291Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:19.405176529Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:19.988878549Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:19.989383423Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:38:20.859426183Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:21.208009861Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:21.208352687Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:21.906915311Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:21.907094379Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:22.77244292Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:23.117091531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:23.117372763Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:25.69861259Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:25.746159675Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:25.746390543Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:26.345066153Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:26.345444172Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:27.383522517Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:27.384071585Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:28.191403862Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:28.192717013Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:28.567745603Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:28.568251373Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:28.942151542Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:38:30.859437667Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:31.770766509Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:31.770986792Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:32.290716929Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:32.291149895Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:32.444898461Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:32.445282274Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:32.492593084Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:32.493153524Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:32.593659082Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:32.594143765Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:32.905479055Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:33.242536186Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:33.243489426Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:33.795331872Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:33.796049486Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:33.816315809Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:33.817450969Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:33.819622082Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:34.151272849Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:34.151510298Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:35.310916006Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:35.577200825Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:35.577692792Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:36.62406847Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:36.624474228Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:36.923390361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:37.170391462Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:37.17084064Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:39.220246752Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:39.220880548Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:40.715167563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:40.715420333Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:38:40.859407016Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:41.244764425Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:41.245165191Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:41.488625642Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:41.510438604Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:41.510860951Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:41.554248179Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:41.554599334Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:41.708968708Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:42.545830948Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:42.546289139Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:43.863830832Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:43.998127603Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:44.338030331Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:45.130718479Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:45.13100657Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:45.358327418Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:45.359085324Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:45.730533765Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:45.731443733Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:45.924180236Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:45.924768109Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:46.375280971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:46.602979089Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:46.603324106Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:46.985235685Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:47.07487094Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:47.075185889Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:48.25842229Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:48.557488072Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:48.647034718Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:48.647480752Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:49.200180205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:49.503824923Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:49.504058978Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:49.658063588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:38:50.859500779Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:51.71139308Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:51.712102337Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:51.891250856Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:51.89179676Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:51.984281336Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:52.088459752Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:52.08886239Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:52.576266437Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:52.69157009Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:52.998765315Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:52.999210329Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:53.303063735Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:53.896364908Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:54.979877526Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:54.980103643Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:55.2958705Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:55.296290395Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:55.413477803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:55.41362694Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:55.647330434Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:55.64778305Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:56.402301112Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:58.307064016Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:58.851752575Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:59.282399028Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:59.282737092Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:59.728215015Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:59.728476147Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:59.921509153Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:38:59.92169124Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:39:00.859429964Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:00.930707034Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:01.114534654Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:01.115006744Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:01.906572112Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:02.345336876Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:02.345494581Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:02.676873373Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:02.677614357Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:03.699654775Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:04.678505939Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:04.72008195Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:07.174235244Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:07.203726486Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:07.22588625Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:07.226167073Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:07.262963788Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:07.263795519Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:08.015934686Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:08.179338491Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:09.486879842Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:09.487285371Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:09.67448404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:09.675332533Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:10.016511426Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:10.435457674Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:39:10.859441764Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:12.43694268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:12.437693089Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:12.597636257Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:14.840410203Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:16.72450545Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:16.725302973Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:17.04507341Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:17.436174287Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:17.436456841Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:18.507214437Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:19.405907143Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:19.934240214Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:19.934761302Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:39:20.859436722Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:21.20841954Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:21.208789371Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:21.920254401Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:21.920413069Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:22.760047857Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:23.127244562Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:23.127524494Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:25.698318161Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:25.744200518Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:25.744466507Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:26.273250041Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:26.27360687Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:27.378613563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:27.378921134Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:28.159150004Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:28.159928522Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:28.616290934Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:28.617125841Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:28.93206546Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:39:30.85952374Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:31.769324584Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:31.769561849Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:31.977095481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:31.977577155Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:32.36054998Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:32.360883372Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:32.497090368Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:32.497605434Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:32.593199096Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:32.593657465Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:32.905502868Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:33.213750023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:33.214070749Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:33.795251913Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:33.796074218Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:33.820144413Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:33.821514488Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:33.824075637Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:34.15040421Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:34.150624833Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:35.311079966Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:35.576587876Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:35.577035994Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:36.61932544Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:36.619884005Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:36.926321104Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:37.160916983Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:37.161341336Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:39.088723985Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:39.089191052Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:40.698834326Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:40.699179728Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:39:40.860299041Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:41.245994394Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:41.246322006Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:41.429486377Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:41.42992486Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:41.486849466Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:41.555772069Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:41.556094352Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:41.709574882Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:42.208622391Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:42.209051475Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:43.86335667Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:43.997903853Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:44.339114275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:45.12835693Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:45.128713348Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:45.35809519Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:45.358835042Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:45.726177967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:45.726896617Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:45.922000269Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:45.92258941Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:46.378258906Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:46.592849452Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:46.593177798Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:46.986280653Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:47.068120033Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:47.068541465Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:48.263210247Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:48.562545114Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:48.64706802Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:48.647656062Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:49.202208799Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:49.557161629Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:49.557410501Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:49.644292901Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:39:50.85942903Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:51.722879878Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:51.723559252Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:51.900603815Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:51.901262283Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:51.986888392Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:52.068325079Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:52.068739612Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:52.576916525Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:52.694136538Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:52.999563864Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:53.000018848Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:53.300549979Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:53.897567276Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:54.984126308Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:54.984355078Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:55.302411743Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:55.30315926Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:55.431133708Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:55.431313936Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:55.852371085Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:55.853027229Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:56.403723564Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:58.309036262Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:58.857761672Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:59.314626962Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:59.314970784Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:59.721892234Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:59.722170472Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:59.886414205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:39:59.886566697Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:40:00.859441598Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:00.93029205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:01.065613456Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:01.066090184Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:02.396190352Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:02.434034038Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:03.882034725Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:04.190715887Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:04.192538137Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:04.651728577Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:04.738926095Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:04.867619582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:08.72841788Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:09.08215809Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:09.084194692Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:09.135770093Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:09.136554963Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:09.866755135Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:09.889495909Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:09.891017067Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:10.19940699Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:10.542354481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:40:10.875821072Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:11.381269623Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:12.137237481Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:12.148159722Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:12.541480927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:12.608484347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:12.609858384Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:12.837740781Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:14.842037701Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:16.726354559Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:16.72682446Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:17.079655823Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:17.527019635Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:17.527267072Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:18.202020607Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:19.457050418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:19.700952526Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:19.701485098Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:40:20.859501988Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:21.90159407Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:21.901792301Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:21.948598905Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:21.94898007Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:22.782256789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:23.116949386Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:23.117220757Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:25.697511279Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:25.74414927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:25.744358452Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:26.292604937Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:26.29291559Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:28.164540234Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:28.166183379Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:28.732136183Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:28.732797332Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:28.962778139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:40:30.859482805Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:31.769707141Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:31.769939056Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:31.955108743Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:31.955521605Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:32.358446422Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:32.3590117Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:32.595298789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:32.595947814Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:32.618911724Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:32.619338842Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:32.905546795Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:33.222621955Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:33.222917838Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:33.795653754Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:33.796431049Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:33.815774373Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:33.816931048Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:33.819038501Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:34.150153923Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:34.150401424Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:35.317081836Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:35.586240971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:35.586762132Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:36.601271026Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:36.601692399Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:36.9255414Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:37.161789548Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:37.162344648Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:37.302610275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:37.302912141Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:39.623798184Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:39.624186857Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:40.687156513Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:40.687410528Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:40:40.859441117Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:41.254181049Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:41.254465314Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:41.460513657Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:41.460865803Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:41.487075657Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:41.55746324Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:41.557767251Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:41.737785371Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:42.549408897Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:42.549857715Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:43.859540987Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:44.00050311Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:44.337710653Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:45.252609112Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:45.252905005Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:45.358229218Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:45.359292989Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:45.731085447Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:45.73180008Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:45.923040194Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:45.923698983Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:46.374259115Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:46.609596161Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:46.609995984Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:46.98575104Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:47.062543522Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:47.062856694Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:48.297161271Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:48.559127232Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:48.673958906Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:48.674522358Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:49.2023635Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:49.566436554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:49.566829748Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:49.64638518Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:40:50.85949691Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:51.712710855Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:51.713702046Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:51.911489235Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:51.912101243Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:51.989819275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:52.075012148Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:52.075417184Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:52.578623102Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:52.691735844Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:52.998857229Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:52.999276164Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:53.314088666Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:53.899277265Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:54.982136065Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:54.982369958Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:55.427443933Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:55.428561947Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:55.452244311Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:55.452415017Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:55.78689907Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:55.787320909Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:56.402678106Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:58.294936643Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:58.85290781Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:59.119503695Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:59.119939721Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:59.711951494Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:59.712244127Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:59.889197769Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:40:59.889344971Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:41:00.859430685Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:00.931988319Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:01.026843802Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:01.027365796Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:01.573272481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:02.345892833Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:02.346045794Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:02.6754862Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:02.676210558Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:03.699328405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:04.681304914Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:04.722403906Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:07.174056969Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:07.205359673Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:07.224046397Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:07.224264462Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:07.272597393Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:07.273598297Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:08.01378911Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:08.847134501Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:08.853337101Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:08.853860688Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:09.737329728Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:09.738003157Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:10.067118964Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:10.432535444Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:41:10.859462159Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:12.444555115Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:12.445231218Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:12.51773473Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:14.843256514Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:16.849758939Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:16.850197217Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:17.074613384Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:17.408259784Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:17.408594815Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:18.117738605Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:19.406846335Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:19.70070765Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:19.701217497Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:41:20.85944674Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:21.208948174Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:21.209269836Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:21.905590658Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:21.905780779Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:22.772808853Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:23.11573485Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:23.115953663Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:25.697051953Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:25.744384091Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:25.744601008Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:26.272020994Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:26.27230789Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:27.381126338Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:27.38140059Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:28.160384461Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:28.160937213Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:28.751062168Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:28.751578323Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:28.936692588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:41:30.859479355Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:31.680215901Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:31.680981183Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:31.770094017Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:31.77050203Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:31.967583339Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:31.968104916Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:32.380699604Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:32.381020452Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:32.593943433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:32.594509441Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:32.905715127Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:33.213933545Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:33.214255109Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:33.79535849Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:33.796132329Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:33.817795345Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:33.819061Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:33.821417427Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:34.160973512Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:34.161275328Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:35.310425783Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:35.576644518Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:35.577188382Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:36.641753333Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:36.642257952Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:36.923920023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:37.160682978Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:37.161107005Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:39.22583577Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:39.2262234Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:40.686021031Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:40.686258156Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:41:40.859493703Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:41.249859206Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:41.250192384Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:41.48725378Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:41.50652575Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:41.506926005Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:41.58330754Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:41.58360308Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:41.722015422Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:42.537398372Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:42.537850412Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:43.85671875Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:43.997573395Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:44.337686737Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:45.181301268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:45.181610062Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:45.358112054Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:45.358913533Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:45.713222279Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:45.713836249Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:45.925578102Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:45.926242557Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:46.37542863Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:46.611476866Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:46.611801378Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:46.985712743Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:47.058455615Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:47.058788332Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:48.284085051Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:48.557703961Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:48.640305271Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:48.641079252Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:49.200953934Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:49.556972756Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:49.557224863Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:49.640622676Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:41:50.859510175Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:51.703053607Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:51.703772911Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:51.908018447Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:51.908502056Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:51.984619742Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:52.070822249Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:52.071222213Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:52.577835524Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:52.691491192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:52.998707051Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:52.999123921Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:53.31420266Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:53.89620663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:54.972575146Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:54.972826189Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:55.333819516Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:55.334272566Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:55.412928522Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:55.413124723Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:55.824548554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:55.825027451Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:56.402636696Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:58.289008306Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:58.855171967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:59.287239492Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:59.287620729Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:59.714086623Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:59.714378019Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:59.880844255Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:41:59.881015481Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:42:00.859417254Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:00.930295395Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:01.002286671Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:01.002781381Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:01.635151396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:02.425654416Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:02.42582303Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:02.712923973Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:02.713853363Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:03.697843215Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:04.6807662Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:04.72022272Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:07.174085596Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:07.205120635Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:07.223323041Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:07.223544045Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:07.264654764Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:07.265395613Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:08.010400374Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:09.358350383Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:09.513547469Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:09.514050988Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:09.71412016Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:09.714838502Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:10.01926173Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:10.434642761Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:42:10.859477626Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:12.427417248Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:12.427994856Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:12.596483778Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:14.849413683Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:16.605432669Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:16.605846052Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:16.842149652Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:16.842550422Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:17.075761444Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:18.27297196Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:19.410717981Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:19.996384605Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:19.996944521Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:42:20.859452363Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:21.207591244Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:21.207966935Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:21.931061699Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:21.931236842Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:22.772801796Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:23.116683772Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:23.116935355Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:25.698419836Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:25.744094588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:25.744314646Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:26.272293115Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:26.27255446Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:27.378554108Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:27.378835473Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:28.167278166Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:28.170263011Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:28.736830874Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:28.737347555Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:28.938803012Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:42:30.859430416Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:31.770278908Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:31.770601725Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:31.91115349Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:31.91165891Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:32.280797965Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:32.281215024Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:32.472485425Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:32.473041399Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:32.594942225Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:32.596466165Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:32.905657457Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:33.219402359Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:33.219754162Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:33.79609278Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:33.797943104Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:33.82519567Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:33.826442722Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:33.828808746Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:34.155634564Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:34.155893443Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:35.312055299Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:35.576854307Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:35.577287397Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:36.628169382Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:36.628645383Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:36.939417805Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:37.160680234Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:37.161125974Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:39.156309994Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:39.156700703Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:40.763029306Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:40.763322034Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:42:40.859534854Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:41.335444794Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:41.335809288Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:41.486929162Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:41.523400924Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:41.52381772Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:41.557947546Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:41.558220566Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:41.715316872Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:42.213392572Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:42.213836346Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:43.859057042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:43.997254228Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:44.337844533Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:44.561720764Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:44.562001391Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:45.35811477Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:45.358952532Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:45.714063295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:45.714764566Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:45.926036499Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:45.926638202Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:46.375743257Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:46.697579439Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:46.697968855Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:46.985486971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:47.088786142Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:47.089121458Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:48.26662522Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:48.520289479Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:48.520787289Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:48.553764027Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:49.203225269Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:49.554645726Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:49.554872156Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:49.653856186Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:42:50.859444013Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:51.70839721Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:51.70911606Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:51.909529226Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:51.910127078Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:51.984259609Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:52.066647835Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:52.067083303Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:52.577468135Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:52.691255354Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:53.006887494Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:53.007446472Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:53.3017807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:53.896340924Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:55.153509481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:55.153769487Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:55.332987183Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:55.333425658Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:55.41743377Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:55.417592245Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:55.742215369Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:55.742623049Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:56.402753279Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:58.284839376Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:58.853970678Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:59.292265931Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:59.292682051Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:59.716142841Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:59.716457178Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:59.891427196Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:42:59.891585031Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:43:00.859437586Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:00.930243405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:01.004602813Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:01.005101014Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:01.798122973Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:02.369101401Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:02.369275244Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:02.740862621Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:02.741567519Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:03.697588924Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:04.685404089Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:04.73190061Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:07.178456656Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:07.210445711Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:07.237822362Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:07.238128297Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:07.265723081Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:07.266407086Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:08.022832488Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:08.104149463Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:09.429289816Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:09.430125555Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:09.695984799Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:09.700090705Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:10.022522456Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:10.433328953Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:43:10.859512647Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:12.441468005Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:12.442070129Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:12.522815368Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:14.859512589Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:16.71472618Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:16.715179653Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:17.074212212Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:17.264888192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:17.265187878Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:18.1494946Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:19.406236668Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:19.695085533Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:19.695630564Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:43:20.859521767Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:21.222297593Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:21.222699688Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:21.903915067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:21.904103948Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:22.759418571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:23.116567813Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:23.116843458Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:25.697115409Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:25.744047213Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:25.744309937Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:26.271928831Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:26.272221694Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:27.385011583Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:27.38526576Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:28.160947131Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:28.161537257Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:28.544922702Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:28.545452883Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:28.941747361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:43:30.859435397Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:31.645573394Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:31.646155467Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:31.770160786Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:31.770413997Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:32.288074286Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:32.288467989Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:32.471621246Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:32.47200086Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:32.593402252Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:32.593855774Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:32.905762873Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:33.233184212Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:33.233511495Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:33.795490166Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:33.796279101Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:33.821487441Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:33.823260482Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:33.826359Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:34.150682327Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:34.150948726Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:35.326934264Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:35.576653638Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:35.577141703Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:36.783236987Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:36.783596779Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:36.929137044Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:37.179878978Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:37.180326307Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:39.158545519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:39.158913773Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:40.732401513Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:40.732818736Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:43:40.860196816Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:41.253730953Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:41.254025839Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:41.487309806Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:41.508599428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:41.508973409Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:41.572147419Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:41.572456014Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:41.724962628Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:42.552944947Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:42.553381397Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:43.857967756Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:43.999047043Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:44.339297636Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:45.137960237Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:45.138286686Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:45.35806273Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:45.358865882Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:45.72630028Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:45.727009506Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:45.915113513Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:45.915708791Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:46.392062988Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:46.607914458Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:46.608223859Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:46.986920284Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:47.058861034Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:47.059182601Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:48.253729637Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:48.512048272Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:48.512631052Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:48.553500567Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:49.200383306Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:49.493613531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:49.493866721Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:49.655789731Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:43:50.859445339Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:51.713288422Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:51.714101946Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:51.920541464Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:51.92113936Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:51.987349682Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:52.105243952Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:52.105796084Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:52.577153512Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:52.692151527Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:52.998835767Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:52.999284959Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:53.300992767Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:53.896708067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:54.979282529Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:54.979524798Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:55.33391812Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:55.334352866Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:55.432437531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:55.432602396Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:55.669028731Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:55.669584265Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:56.405046447Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:58.267550903Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:58.859709155Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:59.289783576Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:59.290078691Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:59.718144562Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:59.718417492Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:59.87058495Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:43:59.870759817Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:44:00.859439504Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:00.930599919Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:00.998876874Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:00.999403586Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:01.581803894Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:02.366934994Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:02.36707606Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:02.693755951Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:02.694380878Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:03.699227351Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:04.677794289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:04.721090971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:07.184832888Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:07.217021652Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:07.235749914Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:07.235988215Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:07.290232111Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:07.291045517Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:08.010811811Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:08.295606379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:09.497586972Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:09.49823511Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:09.69550654Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:09.696527268Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:10.021768471Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:10.430988222Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:44:10.859433785Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:12.465180051Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:12.465793263Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:12.600363439Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:14.841341123Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:16.726337237Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:16.726914041Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:17.07388771Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:17.430387484Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:17.4307987Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:18.384846879Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:19.405514783Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:19.679958276Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:19.680426257Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:44:20.859426964Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:21.209782686Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:21.210114276Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:21.92565213Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:21.927213492Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:22.75756725Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:23.116684728Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:23.116969523Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:25.698935605Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:25.7438967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:25.744184665Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:26.297654085Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:26.297970101Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:27.312468539Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:27.312917434Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:28.159100003Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:28.159879582Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:28.596960011Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:28.597506347Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:28.947677547Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:44:30.859434613Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:31.64711012Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:31.64759734Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:31.771054238Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:31.771300567Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:32.278285844Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:32.278678931Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:32.497243172Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:32.497904872Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:32.601548357Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:32.602292496Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:32.905521953Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:33.2207685Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:33.221044262Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:33.795319575Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:33.796001089Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:33.817947658Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:33.819164944Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:33.821387659Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:34.146634832Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:34.147052619Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:35.310086966Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:35.577018315Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:35.5774319Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:36.609792841Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:36.610223229Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:36.920103613Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:37.161074231Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:37.161522029Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:39.341776943Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:39.342327293Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:40.685652509Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:40.685900598Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:44:40.859426247Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:41.049940598Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:41.050230466Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:41.487829886Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:41.519278347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:41.519742291Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:41.557717477Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:41.558068139Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:41.71807718Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:42.543811654Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:42.54426003Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:43.865887284Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:43.999865481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:44.337603068Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:45.133952572Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:45.13432663Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:45.358101022Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:45.358886151Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:45.73577096Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:45.736379827Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:45.927857251Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:45.928473596Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:46.375309821Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:46.591362361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:46.591721249Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:46.982155157Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:47.058469215Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:47.058821615Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:48.258055124Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:48.554158418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:48.596291307Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:48.598092116Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:49.200588703Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:49.503935408Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:49.504180956Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:49.637390622Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:44:50.859429756Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:51.711851009Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:51.712493703Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:51.891524567Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:51.892081706Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:51.989126801Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:52.086494846Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:52.087020164Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:52.576284481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:52.690975197Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:52.998744904Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:52.999201387Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:53.301149555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:53.896034129Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:54.988530545Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:54.988734252Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:55.305837494Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:55.306452861Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:55.431913334Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:55.4320874Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:55.66938636Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:55.669772695Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:56.402478482Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:58.26831065Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:58.852275762Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:59.309455927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:59.31031742Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:59.721439011Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:59.721744774Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:59.870218681Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:44:59.870347062Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:45:00.859429315Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:00.931267268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:00.994149287Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:00.994641507Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:01.579222408Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:02.569311281Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:02.584946129Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:03.782737405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:03.783933336Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:03.811113498Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:04.694378798Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:04.730861156Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:07.181529741Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:07.211679113Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:07.235164491Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:07.235461289Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:07.278739528Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:07.279563419Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:08.016299262Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:08.345800645Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:09.347259642Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:09.347644841Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:09.73665286Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:09.737452505Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:10.026235519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:10.503357446Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:45:10.859435887Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:12.441438073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:12.442094035Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:12.596821069Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:14.845261936Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:16.842337915Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:16.843335727Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:17.077130045Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:17.456377195Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:17.457163487Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:18.223809484Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:19.409712407Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:19.991798605Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:19.992321992Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:45:20.859417617Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:21.214517349Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:21.214902948Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:21.946658464Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:21.946852486Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:22.777048377Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:23.117879731Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:23.118138804Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:25.697592935Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:25.74470876Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:25.744937702Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:26.341811353Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:26.342128704Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:27.378343945Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:27.378644217Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:28.181491912Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:28.182229086Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:28.525111405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:28.52555656Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:28.939413444Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:45:30.859439358Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:31.620363778Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:31.62090992Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:31.769942891Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:31.77017737Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:32.08076464Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:32.081163172Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:32.460203072Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:32.460573002Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:32.592922537Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:32.593400658Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:32.905533391Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:33.214081398Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:33.214386158Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:33.795487118Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:33.796225269Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:33.816619036Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:33.817789811Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:33.819991481Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:34.132808793Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:34.133049203Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:35.31132869Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:35.586732875Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:35.587162942Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:36.619638815Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:36.620153797Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:36.920560454Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:37.161224485Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:37.161655622Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:39.263878156Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:39.264346598Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:40.685897553Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:40.686148931Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:45:40.859425951Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:41.263087289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:41.26340443Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:41.487109687Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:41.513089764Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:41.513465452Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:41.558514621Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:41.558875578Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:41.708587268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:42.545954406Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:42.546371715Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:43.863805345Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:43.997574458Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:44.338325952Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:45.13376449Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:45.134041662Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:45.358074908Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:45.358823613Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:45.713947152Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:45.714603897Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:45.913412842Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:45.914046715Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:46.378268187Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:46.60476686Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:46.605120257Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:46.983731946Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:47.229354717Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:47.229753877Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:48.257377314Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:48.557608071Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:48.652401088Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:48.653121207Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:49.202827233Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:49.554370449Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:49.554600491Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:49.638821389Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:45:50.859453366Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:51.728321423Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:51.729068338Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:51.92218691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:51.923035263Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:51.987707207Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:52.092080327Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:52.092592971Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:52.577561952Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:52.691456789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:52.998725279Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:52.999188509Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:53.300526036Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:53.89952802Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:54.978117009Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:54.978341092Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:55.305116621Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:55.305532841Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:55.413239189Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:55.413361895Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:55.648135812Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:55.648581583Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:56.402835508Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:58.268309008Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:58.875603189Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:59.288929579Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:59.289228614Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:59.710517194Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:45:59.710981762Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:00.064097607Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:00.064231399Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:46:00.859353128Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:00.930929424Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:01.097690836Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:01.098114471Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:01.542274525Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:02.527851207Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:02.52800893Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:02.750069758Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:02.750731744Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:03.698139045Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:04.686507373Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:04.721150569Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:07.186546089Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:07.216519187Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:07.23690222Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:07.237107983Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:07.2654648Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:07.266276849Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:08.011720017Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:08.56243584Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:09.430377872Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:09.430836231Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:09.727586603Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:09.728933196Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:10.059064487Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:10.445558304Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:46:10.859486173Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:12.438143631Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:12.438756703Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:12.521414444Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:14.840655956Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:16.849704111Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:16.850488771Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:17.074902366Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:17.447629967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:17.448393008Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:18.400906563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:19.409886128Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:19.688772857Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:19.689290919Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:46:20.859428946Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:21.220221873Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:21.220586381Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:21.915488321Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:21.915676514Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:22.760022459Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:23.116775624Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:23.117046956Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:25.698261579Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:25.743915452Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:25.744138091Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:26.271927583Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:26.27220788Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:28.166184186Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:28.166879088Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:28.731274623Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:28.732004271Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:28.939293945Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:46:30.859377298Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:31.625902767Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:31.626438748Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:31.769360093Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:31.7696279Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:31.972755485Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:31.973161011Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:32.352294069Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:32.352695226Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:32.592580712Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:32.593017677Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:32.905558349Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:33.221577816Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:33.221933973Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:33.795286534Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:33.796040167Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:33.8156961Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:33.816837164Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:33.819078507Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:33.848920472Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:33.849230265Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:34.132843832Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:34.133081891Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:35.310297766Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:35.577572347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:35.578037914Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:36.649721483Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:36.650159882Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:36.920701564Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:37.165623493Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:37.166209633Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:39.192530067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:39.192928607Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:40.701097537Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:40.701397859Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:46:40.859428491Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:41.252821813Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:41.253130838Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:41.491396414Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:41.5676198Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:41.568001776Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:41.606410628Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:41.606888829Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:41.756088152Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:42.575043609Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:42.575481582Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:43.863572403Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:43.998775363Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:44.338480416Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:45.122749857Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:45.123067067Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:45.358078992Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:45.358830459Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:45.726593782Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:45.727319971Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:45.924551127Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:45.925134408Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:46.388571102Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:46.756515644Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:46.756903677Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:46.985520675Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:47.221650896Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:47.221984699Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:48.271322412Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:48.553963512Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:48.653272938Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:48.65389036Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:49.20899256Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:49.5578476Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:49.558118665Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:49.650493134Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:46:50.859459708Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:51.703303896Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:51.7042895Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:51.915169247Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:51.915843173Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:51.991241704Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:52.082366904Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:52.082785938Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:52.578009277Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:52.691096362Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:53.010252946Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:53.010716646Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:53.300570007Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:53.898281226Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:54.996146223Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:54.996885603Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:55.296064096Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:55.296537316Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:55.411967418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:55.412112836Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:55.746072588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:55.746484298Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:56.401971163Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:58.344817436Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:58.857372965Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:59.113920018Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:59.114356475Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:59.713738593Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:59.713973104Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:59.886260948Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:46:59.886423331Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:47:00.859438884Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:00.93047219Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:01.017975061Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:01.01847774Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:01.91645792Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:02.360720123Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:02.360882022Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:02.685126688Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:02.68619398Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:03.699597253Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:04.720018818Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:04.771318364Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:07.190413486Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:07.221768928Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:07.249540062Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:07.249904295Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:07.273843964Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:07.27452106Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:08.018951195Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:08.149955663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:09.462824586Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:09.463378199Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:09.734812808Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:09.735368077Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:10.016386606Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:10.43183326Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:47:10.859454226Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:12.446355874Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:12.446981728Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:12.515466888Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:14.844271463Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:16.841731943Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:16.842432509Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:17.072145668Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:17.404172774Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:17.404435458Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:18.519221736Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:19.420532727Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:19.694036216Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:19.694626834Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:47:20.859427502Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:21.2218045Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:21.222133765Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:21.92312891Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:21.923307922Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:22.773541419Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:23.116825099Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:23.117073619Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:25.697907278Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:25.743937466Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:25.74412975Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:26.271934709Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:26.272202122Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:27.329339421Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:27.329577412Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:28.159442023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:28.160982346Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:28.70709629Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:28.707603965Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:28.931836234Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:47:30.859356738Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:31.621186646Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:31.621792508Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:31.769919369Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:31.770151041Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:31.963254916Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:31.96365785Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:32.354643948Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:32.35504965Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:32.593176137Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:32.59360692Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:32.905683065Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:33.213842265Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:33.214190525Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:33.795260393Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:33.796054782Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:33.816199066Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:33.817327196Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:33.819501149Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:34.132549395Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:34.132784255Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:35.312596073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:35.576749811Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:35.577191033Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:36.634772067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:36.635247474Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:36.923366222Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:37.162985159Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:37.163431971Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:39.259249229Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:39.260008658Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:40.685557698Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:40.685833934Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:47:40.85941108Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:41.267817088Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:41.268165114Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:41.429730061Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:41.430071719Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:41.486968179Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:41.576417529Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:41.576715372Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:41.716596832Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:42.554586722Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:42.555059982Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:43.855319244Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:43.999966472Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:44.346444888Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:44.549214739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:44.54953121Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:45.358097669Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:45.358884463Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:45.725047472Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:45.725768599Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:45.950129082Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:45.951100145Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:46.374700117Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:46.603713543Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:46.604067111Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:46.986481749Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:47.226068094Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:47.226381741Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:48.266768539Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:48.553376476Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:48.649572656Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:48.650203998Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:49.200214432Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:49.557040311Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:49.557288108Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:49.655729054Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:47:50.859473638Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:51.699680499Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:51.700333286Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:51.903223878Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:51.903886909Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:51.982431142Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:52.071329101Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:52.07174843Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:52.577037319Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:52.691790429Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:52.998965423Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:52.999440199Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:53.327836165Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:53.896561616Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:54.971961718Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:54.972175948Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:55.304883314Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:55.305343979Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:55.431358685Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:55.431535465Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:55.743706569Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:55.744135494Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:56.403363434Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:58.254335073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:58.854157049Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:59.095054954Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:59.095372392Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:59.713078186Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:59.713337763Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:59.869706547Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:47:59.869859784Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:48:00.859425999Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:00.930296518Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:00.988831244Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:00.989292883Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:01.630600383Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:02.345997776Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:02.346152344Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:02.681227296Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:02.682027982Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:03.697827596Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:04.720534333Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:04.773965162Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:07.165961001Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:07.195214247Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:07.213519328Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:07.213738633Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:07.293625983Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:07.294378359Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:08.011881846Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:08.413703953Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:08.977498721Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:08.978067305Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:09.694206431Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:09.695518937Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:10.022610264Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:10.431314466Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:48:10.859508477Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:12.442199987Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:12.442804531Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:12.597027155Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:14.84148362Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:16.724311305Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:16.725199887Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:17.072331868Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:17.398095353Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:17.398404731Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:18.120363719Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:19.412231584Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:20.074657253Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:20.076039828Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:48:20.859408614Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:21.207200762Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:21.207562864Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:21.931444753Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:21.931639814Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:22.757496202Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:23.116486903Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:23.116782646Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:25.697019082Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:25.928880788Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:25.929086565Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:26.34880302Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:26.34908971Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:27.380426012Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:27.380710655Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:28.203420342Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:28.203936481Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:28.752023212Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:28.752544214Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:28.950639921Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:48:30.859431471Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:31.77501404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:31.775259508Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:32.273777668Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:32.274179992Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:32.349750429Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:32.350137616Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:32.489969207Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:32.490500697Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:32.574655567Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:32.57511086Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:32.905753668Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:33.215041851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:33.215390114Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:33.795309817Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:33.796084784Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:33.81588551Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:33.816975212Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:33.819113004Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:34.132627781Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:34.132920572Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:35.312573125Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:35.577579555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:35.578064228Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:36.610022464Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:36.610428768Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:36.920717336Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:37.170356818Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:37.170812737Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:39.298640052Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:39.299131059Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:40.701093035Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:40.701384605Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:48:40.859440514Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:41.253888616Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:41.254287361Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:41.487085329Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:41.526059358Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:41.526378487Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:41.556231484Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:41.556492077Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:41.719276467Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:42.564844118Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:42.565281991Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:43.857084625Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:44.000891421Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:44.338439392Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:45.115853497Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:45.116205545Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:45.358086027Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:45.358847386Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:45.72076535Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:45.721544455Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:46.008742903Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:46.009367006Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:46.375738862Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:46.604959538Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:46.605300115Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:46.985726208Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:47.077169007Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:47.07755539Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:48.271806968Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:48.554719144Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:48.640178098Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:48.640887616Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:49.200452549Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:49.557036725Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:49.557264999Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:49.730284887Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:48:50.859428581Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:51.709642032Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:51.710372246Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:51.916936096Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:51.917578961Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:51.983345879Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:52.078650517Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:52.079076103Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:52.578018847Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:52.709753486Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:52.999072666Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:52.99954751Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:53.329100269Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:53.898869206Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:54.981038039Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:54.981251538Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:55.332637812Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:55.333107649Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:55.412348372Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:55.412480061Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:55.794582095Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:55.795590026Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:56.403648671Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:58.269872313Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:58.854795392Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:59.279103774Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:59.279367895Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:59.7115532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:59.711817332Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:59.903974767Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:48:59.904131473Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:49:00.859505753Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:00.93074961Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:00.979198554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:00.979615126Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:01.977342681Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:02.347492883Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:02.347685436Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:02.777952969Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:02.778614716Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:03.699610284Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:04.675649217Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:04.719769656Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:07.174223664Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:07.204039238Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:07.223519665Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:07.223835058Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:07.260143446Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:07.261015722Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:08.0400675Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:08.139164367Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:09.049948408Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:09.050350146Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:09.677576082Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:09.678174927Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:10.067898348Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:10.431071406Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:49:10.85942972Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:12.439677941Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:12.440274917Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:12.516457923Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:14.85710889Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:16.724184996Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:16.724601789Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:17.074169917Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:17.401834363Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:17.402114262Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:18.582941695Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:19.507139865Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:19.67948714Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:19.680009781Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:49:20.859467032Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:21.222424335Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:21.222938087Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:21.912227556Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:21.912390374Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:22.773402235Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:23.116098066Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:23.116363083Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:25.702566245Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:25.755481284Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:25.755837275Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:26.271819424Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:26.27213473Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:27.381731851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:27.382004252Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:28.164219848Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:28.164817928Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:28.734567343Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:28.735153231Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:28.947837435Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:49:30.85944533Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:31.769321167Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:31.769572543Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:32.29097119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:32.291410991Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:32.351430289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:32.351810206Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:32.498187937Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:32.498744799Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:32.592798261Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:32.593248281Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:32.90551413Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:33.221027307Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:33.221368328Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:33.79531762Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:33.796073539Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:33.816432103Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:33.817609823Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:33.819834527Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:34.134082107Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:34.134382635Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:35.310111938Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:35.576865352Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:35.57731577Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:36.65247169Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:36.652987471Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:36.920928711Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:37.160732614Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:37.161173695Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:39.375214218Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:39.375729668Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:40.695627752Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:40.695933701Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:49:40.859419739Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:41.260209656Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:41.260510474Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:41.487215603Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:41.523521901Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:41.523927087Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:41.558567558Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:41.558902965Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:41.717555372Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:42.436989099Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:42.437543406Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:43.855615696Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:43.997590583Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:44.272859928Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:45.132633618Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:45.132974283Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:45.358091851Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:45.358910662Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:45.725215328Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:45.725998429Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:45.928996462Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:45.929618687Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:46.377416317Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:46.605846798Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:46.606177393Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:46.982873076Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:47.059458242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:47.059791958Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:48.274500831Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:48.552432596Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:48.646565151Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:48.647262268Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:49.200082756Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:49.493338409Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:49.493581344Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:49.64896573Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:49:50.8595049Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:51.706099487Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:51.706817292Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:51.92513021Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:51.926061845Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:51.983019713Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:52.092576987Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:52.093143792Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:52.577047501Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:52.693629134Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:52.998625866Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:52.999135483Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:53.328532003Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:53.89705649Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:54.979925883Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:54.980143051Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:55.332941295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:55.333373981Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:55.412989654Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:55.413142777Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:55.75783967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:55.758290622Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:56.405649374Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:58.274894928Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:58.852181499Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:59.305187017Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:59.305717618Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:59.717481197Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:59.717794073Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:59.887378145Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:49:59.887521817Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:50:00.859439678Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:00.930139961Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:01.048070367Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:01.048581113Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:02.719621755Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:02.727383584Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:03.30136733Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:03.309425784Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:03.764444847Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:03.806814135Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:04.679517214Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:04.7218331Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:07.208092432Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:07.248022688Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:07.270324257Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:07.27056981Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:07.299576143Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:07.30040482Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:08.01770401Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:08.865369445Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:08.866879676Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:09.769511364Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:09.771014594Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:10.107135468Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:10.46739863Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:10.572017207Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:50:10.859422224Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:12.447775573Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:12.448417466Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:12.518518013Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:14.958352743Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:16.860394206Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:16.861141263Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:17.077524162Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:17.53682371Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:17.537147579Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:18.147727811Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:19.411548447Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:19.684897588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:19.685460423Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:50:20.863018153Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:21.31191935Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:21.313013793Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:21.965553208Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:21.968842149Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:22.777768618Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:23.17016381Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:23.171123756Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:25.698048841Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:25.744786881Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:25.745030317Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:26.271996405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:26.272308759Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:27.351476144Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:27.351895359Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:28.159354291Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:28.160091658Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:28.733886124Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:28.734394388Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:28.964939162Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:50:30.859399695Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:31.643613945Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:31.644214893Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:31.769569736Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:31.769865607Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:32.278966607Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:32.27944132Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:32.462186559Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:32.462528681Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:32.592529595Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:32.592989053Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:32.905540822Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:33.215393405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:33.215712493Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:33.795715248Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:33.796436528Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:33.816923699Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:33.818509805Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:33.821046448Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:34.132839211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:34.133089444Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:35.314244682Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:35.577134963Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:35.577573227Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:36.65185757Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:36.652281823Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:36.922550902Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:37.162293183Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:37.162837828Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:39.59789029Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:39.59837244Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:40.70078078Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:40.701096569Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:50:40.859444697Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:41.249280315Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:41.249557272Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:41.420281399Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:41.42065309Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:41.487039441Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:41.571235243Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:41.571629735Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:41.71771077Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:42.55174093Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:42.552151371Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:43.877935336Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:44.015981067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:44.338140055Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:45.173911581Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:45.174219531Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:45.358105359Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:45.358988273Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:45.731843912Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:45.732648323Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:45.924307397Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:45.92490337Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:46.375128759Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:46.595657167Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:46.596040145Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:46.981461218Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:47.059988458Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:47.060351389Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:48.337826395Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:48.55622073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:48.653808679Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:48.654386166Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:49.204641353Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:49.50814985Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:49.508461794Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:49.645333865Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:50:50.859421013Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:51.720060532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:51.720815502Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:51.903208918Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:51.903771989Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:51.990572136Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:52.083875087Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:52.084295193Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:52.578185886Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:52.714589987Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:52.998820809Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:52.999240042Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:53.32815171Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:53.900990216Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:54.99606865Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:54.99627521Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:55.343099492Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:55.343535408Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:55.434576442Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:55.434893635Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:55.679508936Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:55.680009005Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:56.40449647Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:58.298318728Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:58.862443917Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:59.118996124Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:59.119411388Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:59.716051425Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:59.716305447Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:59.869170821Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:50:59.869320138Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:51:00.859411995Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:00.861382854Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:00.861854232Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:00.931126671Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:02.25769446Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:02.376245971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:02.37640412Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:02.666703994Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:02.667401409Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:03.6993554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:04.675385843Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:04.726179777Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:07.178716249Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:07.210828912Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:07.231174787Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:07.231411729Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:07.314039701Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:07.314783348Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:08.011856675Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:09.760839453Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:09.76122318Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:09.797035473Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:09.813445012Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:09.826961192Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:10.035317868Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:10.435273092Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:51:10.859456641Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:12.44410999Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:12.444736488Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:12.518690772Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:14.849345356Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:16.842063692Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:16.842514483Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:17.078346296Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:17.447952237Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:17.448268755Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:18.189499379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:19.411613076Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:19.99538625Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:19.995882492Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:51:20.859418556Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:21.245390928Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:21.245706552Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:21.906398156Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:21.906576687Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:22.758173732Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:23.115781038Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:23.116064947Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:25.698906697Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:25.744573308Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:25.744888166Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:26.341402881Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:26.341708197Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:27.380575167Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:27.380900529Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:28.15824572Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:28.158756467Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:28.561303655Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:28.561864296Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:28.932135217Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:51:30.859424844Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:31.655714958Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:31.656282502Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:31.769062799Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:31.76928564Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:32.252735904Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:32.253128725Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:32.471919755Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:32.472283259Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:32.59277326Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:32.593214557Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:32.905544844Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:33.219091878Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:33.219439728Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:33.795380727Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:33.796164878Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:33.816236031Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:33.817408994Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:33.819624904Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:34.132413565Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:34.132654713Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:35.30957508Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:35.578171177Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:35.578649933Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:36.619846777Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:36.620333931Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:36.920423951Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:37.161168856Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:37.161577136Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:39.149019021Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:39.151609585Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:40.685734304Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:40.686017713Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:51:40.859422119Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:41.25989462Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:41.26097789Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:41.487750705Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:41.519561369Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:41.519968863Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:41.557325433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:41.557560184Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:41.716534595Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:42.554482403Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:42.554953888Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:43.857207039Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:43.999645728Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:44.274209653Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:45.131756205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:45.1320512Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:45.358059552Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:45.358788646Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:45.77139853Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:45.772131782Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:45.926236516Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:45.926894091Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:46.374781116Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:46.600843406Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:46.601162604Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:46.986038816Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:47.0591565Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:47.059533209Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:48.254620325Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:48.555538875Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:48.659931244Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:48.660760155Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:49.203863235Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:49.559314036Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:49.559565898Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:49.642724466Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:51:50.859430467Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:51.699429534Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:51.700112919Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:51.891527987Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:51.892030484Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:51.985850135Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:52.067689485Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:52.068095777Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:52.577312849Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:52.691728154Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:52.998876579Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:52.999311515Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:53.339506404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:53.897456115Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:54.973808165Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:54.974013425Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:55.317340263Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:55.3181949Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:55.412502999Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:55.412637581Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:55.666517846Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:55.666999478Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:56.403006053Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:58.31131348Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:58.855922663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:59.287617838Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:59.287931505Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:59.714265532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:59.714550427Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:59.889719068Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:51:59.889879558Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:52:00.85950333Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:00.930209255Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:01.09527259Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:01.095784494Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:02.065951889Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:02.367317851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:02.367469688Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:02.678693107Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:02.679505936Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:03.698848739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:04.678346254Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:04.719799984Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:07.183219177Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:07.213890832Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:07.239996134Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:07.240245204Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:07.269294995Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:07.269982801Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:08.010402629Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:08.427497801Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:09.569726345Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:09.571460045Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:09.728095218Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:09.729096164Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:10.019572216Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:10.431770281Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:52:10.859505Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:12.432694744Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:12.433296643Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:12.600338915Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:14.841579011Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:16.604238716Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:16.604534627Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:16.796590709Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:16.797229203Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:17.071948969Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:18.290729853Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:19.409549546Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:19.690729882Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:19.691371333Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:52:20.859556423Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:21.207895392Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:21.208191415Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:21.904090055Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:21.904268799Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:22.761013197Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:23.133783791Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:23.134237274Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:25.697474511Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:25.743847935Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:25.74410262Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:26.272026912Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:26.272326491Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:27.381072734Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:27.381362411Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:28.159189957Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:28.159806572Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:28.569909176Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:28.5704371Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:28.992842386Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:52:30.859425229Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:31.656295382Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:31.656854543Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:31.769032618Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:31.769285762Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:31.962479501Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:31.962887036Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:32.462203191Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:32.46259517Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:32.59252887Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:32.592979956Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:32.905523145Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:33.21380761Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:33.214138221Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:33.795334311Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:33.79633051Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:33.819339876Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:33.820487061Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:33.822683837Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:34.144467845Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:34.144731129Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:35.313619669Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:35.577594186Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:35.578044743Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:36.649532556Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:36.649914621Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:36.966100773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:37.160915292Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:37.161362863Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:39.447074344Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:39.44751443Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:40.68546185Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:40.685763426Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:52:40.859427384Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:41.247020817Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:41.247343888Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:41.487111263Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:41.516068694Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:41.51639737Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:41.554761668Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:41.555035359Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:41.719225684Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:42.212260909Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:42.212740316Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:43.88157503Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:43.998526338Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:44.2931499Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:44.551393949Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:44.551685575Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:45.358247961Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:45.35909524Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:45.73389279Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:45.734697764Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:45.924405453Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:45.925078999Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:46.376358029Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:46.614759912Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:46.615084422Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:46.985692987Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:47.05908618Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:47.059399379Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:48.256759652Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:48.511083371Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:48.511651467Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:48.553350089Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:49.200124757Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:49.556704403Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:49.55696439Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:49.63726183Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:52:50.859418066Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:51.70901977Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:51.709653988Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:51.890711414Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:51.891276779Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:51.983977316Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:52.076911181Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:52.077315413Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:52.578754402Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:52.691633835Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:52.999594979Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:53.000235964Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:53.335341788Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:53.896920335Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:54.984609432Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:54.984857423Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:55.304972596Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:55.305448731Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:55.412797532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:55.412963486Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:55.639279311Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:55.639784225Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:56.403080758Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:58.257349654Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:58.858608879Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:59.289927701Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:59.290266968Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:59.711616367Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:59.711915157Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:59.898304211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:52:59.898486841Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:53:00.859484443Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:00.930732227Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:01.144873221Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:01.145375483Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:02.037791649Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:02.36949013Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:02.369628864Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:02.689159044Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:02.689847506Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:03.698992576Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:04.679543589Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:04.719916571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:07.180047624Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:07.211996217Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:07.233634203Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:07.233932503Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:07.264758949Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:07.265512561Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:08.019155779Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:08.522933043Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:09.475776281Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:09.476085765Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:09.706188277Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:09.706864119Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:10.031233841Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:10.431563366Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:53:10.859428511Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:12.441574943Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:12.44261211Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:12.517175238Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:14.843571231Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:16.714909196Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:16.715344238Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:16.944704404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:16.945049808Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:17.079708525Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:18.135373555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:19.408259567Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:19.687753316Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:19.688272418Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:53:20.859491281Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:21.941898709Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:21.942331792Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:22.013409732Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:22.013818877Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:22.773828623Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:23.126795864Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:23.127056498Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:25.697345571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:25.743817389Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:25.744048492Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:26.273829471Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:26.27410697Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:27.378866989Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:27.379115943Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:28.160526045Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:28.161767551Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:28.537342524Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:28.537810208Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:28.931818914Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:53:30.859417165Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:31.637879575Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:31.638372787Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:31.770104345Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:31.770374256Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:31.956120585Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:31.956487504Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:32.346514291Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:32.346910402Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:32.592316295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:32.592792048Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:32.905483886Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:33.213890207Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:33.214196332Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:33.795249395Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:33.795985701Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:33.815216942Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:33.816338197Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:33.81856589Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:34.132449135Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:34.133273735Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:35.310330443Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:35.577023074Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:35.57749283Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:36.609234181Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:36.609701217Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:36.926215191Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:37.166390716Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:37.166862459Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:39.219199759Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:39.219756568Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:40.685559876Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:40.685850029Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:53:40.859481044Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:41.258909062Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:41.259173138Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:41.487207404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:41.516365484Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:41.516801993Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:41.570082945Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:41.570386307Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:41.707980466Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:42.548640287Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:42.549139289Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:43.870478492Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:43.998817841Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:44.337974979Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:45.118115566Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:45.118456271Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:45.358138462Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:45.35901793Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:45.725866482Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:45.726504509Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:45.917500554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:45.918134153Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:46.374822334Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:46.628493019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:46.628862689Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:46.985950575Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:47.063979866Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:47.064309394Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:48.257798566Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:48.520416234Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:48.520949019Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:48.554164042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:49.201271032Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:49.554815567Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:49.555042303Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:49.647313615Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:53:50.859445452Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:51.703445724Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:51.704149572Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:51.926991109Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:51.927644723Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:52.075976144Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:52.076379903Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:52.082755609Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:52.577109428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:52.697223224Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:53.000310856Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:53.001120395Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:53.335236736Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:53.897105561Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:54.972405603Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:54.972615649Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:55.311123614Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:55.311558761Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:55.437849666Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:55.438007473Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:55.751553491Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:55.752053003Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:56.403402336Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:58.2911608Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:58.855103789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:59.286625604Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:59.286955755Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:59.714678567Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:53:59.714961848Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:00.051377567Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:00.051543708Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:54:00.859481012Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:00.93064824Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:01.025647404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:01.026170952Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:02.243097977Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:02.345560551Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:02.345736285Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:03.561499986Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:03.562190379Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:03.698782746Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:04.677854728Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:04.720101415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:07.178788751Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:07.211889943Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:07.236698629Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:07.23697665Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:07.261463518Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:07.262165316Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:08.024632355Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:08.342785959Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:09.404486324Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:09.404822614Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:09.691391617Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:09.691911371Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:10.015287703Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:10.453338363Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:54:10.860646075Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:12.432997754Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:12.433601606Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:12.519313659Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:14.841583279Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:16.84239465Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:16.842881581Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:17.044270909Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:17.439271338Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:17.439640452Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:18.464339393Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:19.40611939Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:20.012388475Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:20.012967311Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:54:20.859429051Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:21.223514405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:21.223900173Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:21.936019727Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:21.936199469Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:22.76059187Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:23.116588085Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:23.116884427Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:25.697591855Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:25.745786906Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:25.745992938Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:26.342743621Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:26.343021959Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:27.31542543Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:27.315778039Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:28.159458275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:28.160527235Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:28.836895314Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:28.837417451Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:28.93204211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:54:30.859496489Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:31.627266162Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:31.627807682Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:31.770036633Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:31.770251084Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:32.281467358Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:32.281887763Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:32.378095424Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:32.378410028Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:32.592427306Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:32.592853647Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:32.905518488Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:33.226539051Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:33.227199477Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:33.795740149Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:33.796434026Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:33.815807416Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:33.816963179Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:33.819162593Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:34.132969608Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:34.133235329Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:35.311366059Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:35.577008023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:35.577499078Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:36.597010555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:36.597402673Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:36.924547725Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:37.1611004Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:37.161573862Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:39.262875353Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:39.263337318Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:40.685696243Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:40.685936556Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:54:40.85946284Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:41.261091183Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:41.261407412Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:41.486693315Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:41.508957556Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:41.509299875Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:41.555146809Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:41.555483946Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:41.715932167Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:42.54762643Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:42.548092975Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:43.955005466Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:43.99817517Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:44.338320843Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:45.124170982Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:45.124462839Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:45.358190318Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:45.359059076Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:45.715450687Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:45.716129642Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:45.92543036Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:45.926228775Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:46.375209443Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:46.603422831Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:46.603794923Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:46.982909462Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:47.079152686Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:47.079486518Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:48.277342331Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:48.510911804Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:48.511382817Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:48.554189737Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:49.200658797Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:49.555468144Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:49.555743209Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:49.664972038Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:54:50.85944396Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:51.709837938Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:51.710545688Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:51.904034797Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:51.904574005Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:51.983093548Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:52.098061516Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:52.098474317Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:52.577350907Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:52.690959099Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:52.998581525Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:52.999028416Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:53.300759302Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:53.898139991Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:54.997777199Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:54.9979774Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:55.296542727Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:55.29706602Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:55.41238665Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:55.41269268Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:55.766984696Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:55.767385036Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:56.402374791Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:58.314857651Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:58.853459609Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:59.294477853Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:59.294822881Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:59.713582286Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:59.713901457Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:59.891123112Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:54:59.891284217Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:55:00.859452174Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:00.930149218Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:01.022781773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:01.023265216Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:02.460561482Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:02.508975264Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:02.543763976Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:02.981223358Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:02.98505482Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:03.698376222Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:04.688378537Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:04.720134434Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:07.193716698Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:07.22543549Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:07.245444749Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:07.245744864Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:07.259067608Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:07.259820469Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:08.01149801Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:08.376714569Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:09.283325541Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:09.283691006Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:09.691226657Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:09.691823974Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:10.018409185Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:10.43837921Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:55:10.859516023Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:12.436984477Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:12.437551633Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:12.517746682Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:14.841012846Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:16.841810914Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:16.842416002Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:17.076140096Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:17.441352829Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:17.441690767Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:18.196263557Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:19.407994678Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:19.69183305Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:19.692319611Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:55:20.859499074Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:21.205985907Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:21.20635719Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:21.934271131Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:21.934455698Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:22.774174114Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:23.126725334Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:23.126992878Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:25.703152974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:25.745022083Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:25.745312714Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:26.296509117Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:26.29679291Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:27.380523474Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:27.380777395Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:28.167361044Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:28.167932517Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:28.738049067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:28.738554216Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:28.937198082Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:55:30.859438052Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:31.635144355Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:31.635734914Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:31.769356833Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:31.769583748Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:32.28630268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:32.286736998Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:32.344055987Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:32.344403685Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:32.592991328Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:32.593423358Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:32.907575878Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:33.213574061Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:33.213861759Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:33.795235077Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:33.795996238Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:33.816690365Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:33.817842777Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:33.820033135Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:34.133226524Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:34.133481067Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:35.319182706Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:35.576853588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:35.577283886Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:36.620369368Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:36.620896763Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:36.921759386Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:37.160634922Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:37.161099239Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:39.205595864Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:39.206241913Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:40.700889037Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:40.701140143Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:55:40.859426626Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:41.270443739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:41.270807054Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:41.486692318Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:41.526896451Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:41.527220401Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:41.55773087Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:41.558016628Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:41.70880058Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:42.549309853Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:42.54978687Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:43.856838728Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:43.999034739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:44.338556291Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:45.135158463Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:45.135515906Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:45.358173134Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:45.358940053Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:45.735104259Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:45.735849364Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:45.916225547Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:45.916823608Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:46.37587741Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:46.589147045Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:46.589470348Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:46.981094487Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:47.071505545Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:47.071860324Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:48.259610392Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:48.557442353Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:48.648112792Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:48.648742112Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:49.204985286Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:49.503635301Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:49.503896648Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:49.741730792Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:55:50.859489372Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:51.712816484Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:51.713576334Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:51.928298364Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:51.929289222Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:51.987523298Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:52.077076228Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:52.077483216Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:52.577651426Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:52.69120641Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:52.998909608Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:52.999330734Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:53.301576413Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:53.897634103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:54.974137711Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:54.97437027Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:55.332698084Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:55.333166194Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:55.489192542Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:55.489471059Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:55.806934806Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:55.807413532Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:56.412717902Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:58.301899268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:58.858302049Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:59.294997723Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:59.295288595Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:59.715161118Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:59.715431036Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:59.887162724Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:55:59.887336891Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:56:00.859411601Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:00.931868805Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:01.039758331Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:01.04029985Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:01.496154921Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:02.346325778Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:02.346490585Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:02.676259285Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:02.676967636Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:03.697648838Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:04.678713993Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:04.720503164Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:07.183629845Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:07.216970238Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:07.2447087Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:07.245009591Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:07.270519922Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:07.271226294Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:08.012363692Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:08.696284129Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:09.463930378Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:09.464355837Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:09.751548359Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:09.753751902Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:10.062275382Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:10.431208038Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:56:10.859454936Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:12.438053324Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:12.438643314Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:12.596332333Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:14.840867956Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:16.721731613Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:16.722172268Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:17.073894481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:17.45016703Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:17.450563303Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:18.223913826Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:19.407065069Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:19.698595898Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:19.699121261Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:56:20.85950889Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:21.210318281Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:21.210707921Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:21.914347717Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:21.914537347Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:22.773231838Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:23.11647011Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:23.116776436Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:25.704022063Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:25.74584016Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:25.74627896Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:26.27644762Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:26.277218217Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:27.383594401Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:27.383898481Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:28.161655179Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:28.162237162Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:28.69227161Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:28.692625481Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:28.94733426Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:56:30.859432453Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:31.770630792Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:31.770879144Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:32.279261193Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:32.279715131Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:32.345967169Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:32.346355079Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:32.514005499Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:32.51450132Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:32.593835082Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:32.594232921Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:32.905611298Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:33.213980718Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:33.214359413Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:33.795263865Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:33.79589615Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:33.816524183Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:33.817785868Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:33.820003186Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:34.132402483Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:34.132635966Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:35.311880082Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:35.59024562Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:35.590721805Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:36.603584143Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:36.603963478Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:36.92017162Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:37.160975243Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:37.16143686Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:39.334201856Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:39.336708002Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:40.685591092Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:40.685872509Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:56:40.859407089Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:41.048411287Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:41.048749704Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:41.487736255Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:41.528797949Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:41.52920357Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:41.558892584Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:41.559225109Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:41.71842043Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:42.537638316Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:42.538087365Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:43.863236068Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:43.9986915Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:44.338154409Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:45.133779319Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:45.134082571Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:45.358098114Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:45.358824561Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:45.726215923Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:45.726885063Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:45.924896716Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:45.925494244Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:46.3741083Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:46.762070763Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:46.762436901Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:46.985122039Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:47.075787583Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:47.076058964Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:48.255854Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:48.555077959Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:48.651931645Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:48.652652303Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:49.202684853Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:49.492768651Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:49.492996922Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:49.644884964Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:56:50.859423922Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:51.717990036Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:51.718984154Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:51.982055284Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:52.008213484Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:52.009728686Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:52.142765443Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:52.143262589Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:52.578775588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:52.691048568Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:52.998939683Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:52.999387504Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:53.315564797Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:53.896270873Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:54.984405137Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:54.984621599Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:55.340196592Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:55.340718705Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:55.431004244Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:55.431161058Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:55.747337468Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:55.747757671Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:56.402470992Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:58.281403636Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:58.852997447Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:59.082011757Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:59.082322342Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:59.712846341Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:59.713115398Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:59.89795545Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:56:59.898102746Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:57:00.85943679Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:00.930961456Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:01.062887221Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:01.06340661Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:02.254328395Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:02.378724589Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:02.378905671Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:02.668809415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:02.669493005Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:03.698530534Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:04.68020726Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:04.720554963Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:07.195695382Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:07.227233591Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:07.245959371Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:07.246197795Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:07.259288158Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:07.260215403Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:08.01826141Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:08.093144206Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:08.894235968Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:08.894562555Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:09.695338637Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:09.696493185Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:10.015904541Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:10.430776727Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:57:10.859441703Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:12.433651696Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:12.434439636Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:12.52107688Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:14.849206668Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:16.714429433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:16.714911924Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:17.074889451Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:17.443598039Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:17.44395957Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:18.151146725Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:19.408262948Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:19.995187535Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:19.995724524Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:57:20.859547529Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:21.224902704Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:21.225217289Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:21.922345982Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:21.922554884Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:22.760312627Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:23.126693913Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:23.126932305Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:25.698001922Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:25.744217014Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:25.744453001Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:26.352919749Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:26.353235686Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:27.328096012Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:27.328360387Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:28.160334966Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:28.16103254Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:28.789362266Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:28.790539197Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:28.937629006Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:57:30.859457668Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:31.770042075Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:31.770305712Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:32.291327362Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:32.291757458Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:32.446687191Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:32.447041706Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:32.504108253Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:32.504658188Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:32.593374591Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:32.593862858Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:32.905923567Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:33.222040427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:33.222320635Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:33.79524437Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:33.795996755Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:33.816085854Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:33.817225902Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:33.819427048Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:34.133235093Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:34.133492877Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:35.31251254Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:35.576887236Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:35.577357646Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:36.647092195Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:36.647569272Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:36.921755415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:37.165539307Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:37.166087407Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:39.345704065Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:39.346163214Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:40.701981804Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:40.702216849Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:57:40.859506258Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:41.247167584Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:41.247482554Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:41.488488918Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:41.511447177Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:41.511864953Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:41.554280547Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:41.554588936Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:41.724425433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:42.559336989Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:42.559778359Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:43.858715668Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:43.999009489Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:44.339447648Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:44.565279959Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:44.565560981Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:45.358028502Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:45.358812508Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:45.726643124Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:45.727368489Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:45.922962861Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:45.923657718Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:46.375818991Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:46.60372945Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:46.604082141Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:46.985904229Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:47.212742206Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:47.213048579Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:48.2605796Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:48.553720465Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:48.639907052Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:48.640337628Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:49.200578923Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:49.554408863Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:49.554699828Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:49.637228776Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:57:50.859441163Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:51.70011837Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:51.70079489Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:51.906276734Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:51.90690801Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:52.072340637Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:52.072793305Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:52.081004033Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:52.577714404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:52.692334037Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:53.008750588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:53.009204375Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:53.301221472Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:53.897262306Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:54.97408961Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:54.974305275Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:55.333356448Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:55.333960261Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:55.433221872Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:55.433371684Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:55.751082435Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:55.751518794Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:56.403620359Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:58.29340898Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:58.851432986Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:59.134081211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:59.13447059Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:59.715628928Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:59.715945747Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:59.870389147Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:57:59.870550512Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:58:00.859433555Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:00.930237117Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:01.004117278Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:01.004620393Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:02.295137547Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:02.383227234Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:02.383385349Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:02.69546115Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:02.696401188Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:03.698922929Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:04.679378368Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:04.720500694Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:07.190220244Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:07.222412846Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:07.248252817Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:07.248496628Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:07.274198269Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:07.2749058Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:08.011555259Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:08.248655418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:08.973650803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:08.974033613Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:09.697972552Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:09.698450529Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:10.01254706Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:10.4304759Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:58:10.859476973Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:12.447419789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:12.448231717Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:12.520563477Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:14.843631501Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:16.846138612Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:16.847456914Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:17.075285328Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:17.409034958Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:17.409388635Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:18.342539757Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:19.403412954Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:19.68664716Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:19.687165519Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:58:20.859461712Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:21.208479488Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:21.208822636Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:21.938520473Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:21.939487982Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:22.774974512Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:23.116836457Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:23.117100321Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:25.698479779Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:25.74395396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:25.744185519Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:26.272030696Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:26.272360473Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:27.380869627Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:27.381366798Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:28.166282806Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:28.167481431Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:28.770444286Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:28.771571091Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:28.961441952Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:58:30.859394426Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:31.639950889Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:31.640500491Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:31.769208712Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:31.769480303Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:31.96416653Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:31.964565268Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:32.471067311Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:32.471536746Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:32.592427142Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:32.592938347Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:32.905507588Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:33.214019557Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:33.214322704Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:33.795238868Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:33.795964948Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:33.815307376Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:33.816434666Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:33.818555028Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:34.133343691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:34.133657219Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:35.31153801Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:35.578148994Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:35.578562037Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:36.601283218Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:36.601692509Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:36.920328561Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:37.168823032Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:37.169243056Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:39.216234325Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:39.216740667Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:40.70174745Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:40.701991434Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:58:40.85942996Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:41.244954559Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:41.245232895Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:41.488598052Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:41.506347512Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:41.506701816Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:41.554540542Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:41.554936783Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:41.745266054Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:42.553396839Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:42.55386136Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:43.859906308Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:43.997817139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:44.3375083Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:45.133958218Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:45.134260854Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:45.358102073Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:45.358836145Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:45.717808417Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:45.718455455Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:45.918726151Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:45.919310846Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:46.392804991Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:46.625857153Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:46.626465015Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:46.983580692Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:47.223365886Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:47.223728687Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:48.273081996Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:48.556194761Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:48.653402954Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:48.654186242Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:49.200113132Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:49.554819379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:49.555076552Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:49.64158393Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:58:50.859511449Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:51.712497798Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:51.713255351Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:51.894990853Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:51.895538537Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:51.986868682Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:52.075638452Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:52.076164017Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:52.57828612Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:52.690963624Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:53.000435912Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:53.001064832Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:53.302329986Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:53.897834258Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:54.970767539Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:54.970949843Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:55.332446998Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:55.332893336Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:55.437375977Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:55.437736854Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:55.648296381Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:55.648737916Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:56.403543207Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:58.294244211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:58.854129069Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:59.28927215Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:59.289606277Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:59.710588904Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:59.710877249Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:59.870266289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:58:59.870414475Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:59:00.859427961Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:00.930782194Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:01.066648169Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:01.067303463Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:01.83886871Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:02.370447405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:02.370604126Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:02.696744892Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:02.697464308Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:03.69722658Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:04.678631924Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:04.719587046Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:07.177349477Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:07.207402427Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:07.226617881Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:07.226897548Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:07.270908005Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:07.271685016Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:08.014448334Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:08.284582613Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:09.426443633Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:09.427216717Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:09.679925159Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:09.680365229Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:10.04245598Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:10.431730621Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:59:10.859438388Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:12.441742589Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:12.442307107Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:12.595803536Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:14.84731107Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:16.842169996Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:16.842633135Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:17.080150231Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:17.424136924Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:17.424596746Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:18.346150378Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:19.406246207Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:19.703461226Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:19.704002319Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:59:20.859430169Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:21.215747182Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:21.216079467Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:21.898604837Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:21.898808385Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:22.773947246Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:23.116084232Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:23.116334781Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:25.697577159Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:25.762763017Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:25.763001187Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:26.272105986Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:26.272373331Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:27.381704309Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:27.382003512Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:28.174465001Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:28.17513359Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:28.570340905Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:28.571092512Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:28.933723328Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:59:30.859460721Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:31.653450292Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:31.654021135Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:31.772688531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:31.772946392Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:31.962120842Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:31.962577947Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:32.489519088Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:32.490040533Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:32.602080097Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:32.602536256Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:32.905520605Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:33.21443496Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:33.214776549Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:33.795262696Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:33.796074813Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:33.816215776Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:33.817389054Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:33.819627997Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:34.133288745Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:34.133543999Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:35.312742018Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:35.57674638Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:35.577199005Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:36.625600792Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:36.626333868Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:36.92232525Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:37.160644261Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:37.161117636Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:39.094539635Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:39.094941941Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:40.770438794Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:40.770769747Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:59:40.860286149Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:41.231006281Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:41.231308616Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:41.496099601Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:41.516811593Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:41.517142191Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:41.559950118Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:41.560261778Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:41.717409865Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:42.225139401Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:42.225562214Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:43.857058019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:43.998342456Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:44.275037263Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:44.551351531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:44.551634977Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:45.358029192Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:45.35879003Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:45.735035994Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:45.735754209Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:45.946750053Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:45.947341182Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:46.375183792Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:46.607139664Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:46.607475288Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:46.982982189Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:47.216475123Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:47.216768378Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:48.262144242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:48.554154527Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:48.650576178Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:48.651342292Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:49.200119685Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:49.562573364Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:49.562890297Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:49.637127712Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T06:59:50.859449507Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:51.707828153Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:51.708514415Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:51.937358448Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:51.937874146Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:51.982459333Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:52.078571767Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:52.079049955Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:52.577655642Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:52.69161122Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:52.998896498Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:52.999386427Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:53.301306119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:53.897857216Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:54.979128444Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:54.97935551Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:55.304847444Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:55.30529403Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:55.475762571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:55.475954204Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:55.676877694Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:55.677300928Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:56.402583188Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:58.299484704Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:58.853287521Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:59.301565452Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:59.302080941Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:59.714015531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:59.71428395Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:59.870297096Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T06:59:59.870442004Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:00:00.859422183Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:00.930270018Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:01.162792212Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:01.163264204Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:02.966729028Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:03.110551136Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:03.84048093Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:04.737916071Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:05.064117747Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:05.622262686Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:05.699093772Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:07.306308009Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:07.307570365Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:07.313377825Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:07.401023653Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:07.438153336Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:07.458791992Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:07.45904386Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:08.144758658Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:09.675145145Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:09.675616859Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:10.012387947Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:10.209805288Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:10.210222708Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:10.437126611Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:00:10.859426318Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:10.928120447Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:12.487056561Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:12.487772027Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:12.58575938Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:14.869126328Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:16.617908508Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:16.618509618Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:16.715213003Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:16.715722868Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:17.086707545Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:18.150008599Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:19.409130687Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:19.998013715Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:19.998540082Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:00:20.85944209Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:21.220074283Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:21.220388651Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:21.920956792Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:21.921117072Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:22.773124832Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:23.118819228Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:23.119116456Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:25.698245523Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:25.744547069Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:25.744851448Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:26.342364541Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:26.342717063Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:27.381458846Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:27.382791568Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:28.159452794Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:28.15997871Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:28.730543539Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:28.733058305Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:28.965338336Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:00:30.865823315Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:31.658940128Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:31.659480927Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:31.769940285Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:31.770171086Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:32.268192675Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:32.268595099Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:32.481822167Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:32.482176798Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:32.573269064Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:32.573746639Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:32.905445741Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:33.222018839Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:33.222363781Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:33.795429089Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:33.796177795Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:33.816895634Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:33.818053274Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:33.820260094Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:34.132542213Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:34.132786014Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:35.313163961Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:35.576881511Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:35.577334572Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:36.65918276Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:36.659586037Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:36.927313995Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:37.160844903Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:37.161297822Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:39.569807934Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:39.570285891Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:00:40.8594307Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:41.254998847Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:41.255318512Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:41.437790114Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:41.438285453Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:41.486825548Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:41.558509545Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:41.558966905Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:41.623801863Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:41.624115514Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:41.709212409Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:42.560460592Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:42.560904679Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:43.869736096Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:44.000179758Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:44.338759404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:45.184157567Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:45.184454648Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:45.358231176Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:45.358994997Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:45.749307741Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:45.750259536Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:45.92253991Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:45.923480073Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:46.378220911Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:46.723348629Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:46.72377451Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:46.981450532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:47.060853913Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:47.061118383Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:48.253945086Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:48.554796625Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:48.657588563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:48.667893691Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:49.204856165Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:49.556835242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:49.557088365Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:49.646769792Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:00:50.859451355Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:51.809462909Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:51.81092088Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:51.937957973Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:51.938514478Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:51.982741462Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:52.130923626Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:52.131446991Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:52.579115882Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:52.692075456Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:52.998994688Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:52.999432077Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:53.305627201Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:53.89724928Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:55.179391265Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:55.179604903Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:55.297282235Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:55.297733922Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:55.476054476Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:55.476217957Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:55.77098287Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:55.771346538Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:56.40789202Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:58.285007101Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:58.85518682Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:59.144635844Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:59.148531822Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:59.725797033Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:59.726075549Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:59.869816806Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:00:59.869947726Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:01:00.85943138Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:00.933431967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:00.941845188Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:00.942430427Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:02.343824971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:02.378519867Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:02.378748006Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:02.674871526Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:02.675562873Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:03.698029331Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:04.6793919Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:04.779990129Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:07.174126374Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:07.204463604Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:07.223744559Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:07.224034522Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:07.278726964Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:07.279429756Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:08.013648317Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:09.413695736Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:09.413992715Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:09.691787589Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:09.69229737Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:10.009936609Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:10.435029394Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:01:10.859391345Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:11.742836704Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:12.443286452Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:12.443906654Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:12.517460146Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:14.851059571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:16.714964467Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:16.715421145Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:17.076656661Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:17.421356725Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:17.421630924Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:18.310041392Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:19.414457863Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:19.693500737Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:19.694062482Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:01:20.859439542Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:21.227619892Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:21.227960157Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:21.977353317Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:21.977511882Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:22.773298589Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:23.182633238Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:23.182916055Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:25.697379844Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:25.744002103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:25.744242613Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:26.2932569Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:26.29359513Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:27.380356842Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:27.380633192Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:28.174991266Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:28.175899117Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:28.823886513Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:28.824417598Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:28.933420122Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:01:30.859432733Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:31.652877311Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:31.653440674Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:31.769617532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:31.76989396Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:32.279251765Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:32.279645755Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:32.350721558Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:32.351117831Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:32.59308949Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:32.593565167Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:32.905480874Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:33.216469931Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:33.220624007Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:33.795272244Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:33.796011404Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:33.815587469Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:33.816749296Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:33.818893875Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:34.132503423Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:34.132795534Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:35.315292012Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:35.576650257Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:35.57708603Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:36.704011873Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:36.70452094Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:36.921877912Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:37.161089883Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:37.161523208Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:39.374323954Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:39.375030012Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:40.70416213Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:40.704482777Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:01:40.859404365Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:41.251119472Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:41.251434783Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:41.463003362Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:41.463395172Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:41.490288396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:41.565810701Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:41.566132743Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:41.72198351Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:42.545111284Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:42.545636867Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:43.857732098Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:44.002266595Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:44.27085422Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:45.130054694Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:45.130377333Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:45.358054277Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:45.35883861Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:45.71923557Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:45.719965297Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:45.918000731Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:45.918609752Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:46.37470773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:46.603850975Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:46.604188007Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:46.986252585Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:47.075591541Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:47.075966759Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:48.25727705Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:48.554245638Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:48.672773394Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:48.673234102Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:49.201005877Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:49.503853391Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:49.504083663Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:49.649144461Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:01:50.859336905Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:51.750179864Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:51.7509154Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:51.923600597Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:51.924708577Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:52.084947249Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:52.108347934Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:52.108917759Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:52.577238209Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:52.693528697Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:52.998500281Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:52.998977587Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:53.301677122Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:53.896307983Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:54.972384099Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:54.972607575Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:55.305634918Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:55.306162923Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:55.41384662Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:55.413995406Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:55.7558006Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:55.756195057Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:56.407646598Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:58.26604207Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:58.852266963Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:59.295116725Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:59.295373828Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:59.723045459Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:59.723326953Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:59.869979132Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:01:59.87010935Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:02:00.859447666Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:00.930623884Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:01.073179194Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:01.073656599Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:01.623340446Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:02.369317972Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:02.369478956Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:02.698416501Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:02.699096548Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:03.700188953Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:04.681277768Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:04.71999465Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:07.195324214Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:07.23241104Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:07.258105207Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:07.258368225Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:07.27892563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:07.279607202Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:08.012985459Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:08.391639002Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:09.44617377Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:09.446589595Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:09.704882613Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:09.705395381Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:10.012597896Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:10.430717169Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:02:10.859452897Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:12.430540439Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:12.431124617Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:12.596538962Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:14.840848463Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:16.615414578Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:16.615732497Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:16.847098493Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:16.847534277Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:17.072979492Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:18.234229461Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:19.407973209Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:19.698722048Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:19.699237537Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:02:20.859423343Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:21.208426315Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:21.208790753Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:21.922854634Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:21.923044982Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:22.772749312Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:23.118348891Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:23.118575942Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:25.697436005Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:25.743910357Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:25.744141425Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:26.271735314Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:26.272054139Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:27.381158617Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:27.381440376Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:28.159152279Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:28.159611666Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:28.728008003Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:28.728527985Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:28.942472677Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:02:30.85941605Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:31.732694365Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:31.734010618Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:31.774073217Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:31.774506805Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:32.307612971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:32.308571454Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:32.454176725Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:32.455099915Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:32.602412513Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:32.603264259Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:32.906754618Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:33.217008835Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:33.217278488Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:33.795324072Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:33.796072229Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:33.815542467Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:33.816652911Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:33.818779184Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:34.133263092Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:34.133517066Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:35.322632797Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:35.576918096Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:35.577360052Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:36.60834721Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:36.608741761Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:36.920569375Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:37.160967624Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:37.161386332Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:39.231061713Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:39.231529422Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:40.687122242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:40.687440523Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:02:40.859496965Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:41.272749615Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:41.273070596Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:41.487222595Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:41.511047254Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:41.511395019Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:41.572347219Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:41.572595228Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:41.695754091Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:42.208285859Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:42.208765449Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:43.857454433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:43.99753151Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:44.276521085Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:44.573721757Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:44.574068906Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:45.358135035Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:45.35889171Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:45.732653994Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:45.733337466Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:45.926895464Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:45.927477225Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:46.375980513Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:46.603631921Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:46.604025856Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:46.985560222Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:47.059947631Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:47.060277862Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:48.256335782Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:48.519932437Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:48.520408653Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:48.558371291Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:49.201896567Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:49.496593798Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:49.496849637Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:49.643759557Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:02:50.859454381Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:51.701051174Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:51.701748139Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:51.8932161Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:51.893716564Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:51.98140134Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:52.075978411Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:52.076332954Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:52.577458025Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:52.692472008Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:52.999301624Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:52.999758653Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:53.315729347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:53.897222754Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:54.977016459Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:54.977239765Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:55.335837602Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:55.336287143Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:55.412544575Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:55.412715768Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:55.754902073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:55.755308321Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:56.402590061Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:58.254997155Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:58.859347457Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:59.293219585Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:59.293561884Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:59.714267611Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:02:59.714589256Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:00.057119864Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:00.057291642Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:03:00.85942483Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:00.930135627Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:01.062890535Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:01.063360646Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:01.69859739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:02.345318024Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:02.345477852Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:02.681161919Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:02.682257418Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:03.699134492Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:04.685001668Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:04.719906555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:07.185122477Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:07.21524765Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:07.241455434Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:07.241763625Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:07.269101542Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:07.269804422Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:08.011602443Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:08.133238449Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:09.419459082Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:09.419784542Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:09.697022557Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:09.697608352Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:10.016497423Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:10.441912971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:03:10.863649273Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:12.440702505Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:12.441320521Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:12.518853538Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:14.844215155Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:16.848439003Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:16.849128006Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:16.957985639Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:16.958337605Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:17.073068778Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:18.14774433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:19.405245149Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:20.003042788Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:20.003598658Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:03:20.859437222Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:21.219322076Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:21.219650345Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:21.913120126Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:21.913297639Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:22.772824088Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:23.11633452Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:23.116629779Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:25.699080583Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:25.744628149Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:25.74496201Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:26.339818003Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:26.340082899Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:28.164037405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:28.164622514Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:28.760981321Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:28.76149854Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:28.938450081Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:03:30.859430296Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:31.635872028Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:31.636438892Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:31.769506872Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:31.769797837Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:32.279627479Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:32.280048738Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:32.348811805Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:32.349247469Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:32.592807682Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:32.593262865Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:32.90555422Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:33.21433399Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:33.214675832Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:33.795256353Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:33.796085785Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:33.816611742Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:33.817825637Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:33.820021669Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:34.14255141Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:34.142858055Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:35.312725013Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:35.576813999Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:35.577297685Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:36.646106466Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:36.646499536Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:36.925078099Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:37.168293373Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:37.16875827Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:37.302632364Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:37.302950912Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:39.24683178Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:39.247237495Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:40.687369954Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:40.687654286Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:03:40.859440193Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:41.250947103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:41.251277063Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:41.487501925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:41.519054811Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:41.519557606Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:41.558290372Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:41.558558845Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:41.71644135Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:42.570753506Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:42.571225851Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:43.859908009Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:43.998489633Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:44.339738042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:45.126043052Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:45.126367241Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:45.35830831Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:45.359076113Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:45.726592503Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:45.727264371Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:45.926470921Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:45.927097499Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:46.37528588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:46.66773297Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:46.668060974Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:46.98683426Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:47.058935177Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:47.059251285Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:48.256291435Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:48.512026551Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:48.512612558Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:48.5541699Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:49.200046832Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:49.493074514Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:49.493273313Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:49.643832668Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:03:50.85943453Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:51.724820922Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:51.725482763Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:51.902217118Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:51.90491203Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:51.987466103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:52.068110247Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:52.068581569Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:52.576929625Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:52.694940377Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:52.999017457Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:52.999405812Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:53.339591317Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:53.897919951Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:54.971540704Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:54.971788912Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:55.332990179Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:55.333479062Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:55.476047399Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:55.476226986Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:55.762451674Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:55.76294071Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:56.402113932Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:58.265721434Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:58.857127695Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:59.294181279Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:59.294569725Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:59.717612239Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:59.717902296Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:59.89022842Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:03:59.89038308Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:04:00.859416525Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:00.930909527Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:00.952091275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:00.952496491Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:01.934596564Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:02.347241942Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:02.347875195Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:02.722302443Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:02.723276932Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:03.698777902Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:04.679164506Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:04.720054454Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:07.187628855Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:07.218368823Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:07.238069794Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:07.23831816Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:07.325707855Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:07.326806258Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:08.015620921Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:09.373096257Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:09.414875098Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:09.41523713Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:09.695421784Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:09.696073189Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:10.041184907Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:10.431198948Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:04:10.859489445Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:12.432026448Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:12.432643093Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:12.595945666Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:14.840083091Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:16.716141463Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:16.716590927Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:17.074175681Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:17.426983361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:17.42732392Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:18.298843854Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:19.410818715Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:19.684130751Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:19.68464806Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:04:20.859431541Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:21.20708635Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:21.207384762Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:21.920461709Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:21.920632772Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:22.77380948Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:23.135288673Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:23.135612897Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:25.697945422Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:25.743948523Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:25.744179321Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:26.271928555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:26.272212236Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:27.314345077Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:27.314640081Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:28.159417632Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:28.160247543Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:28.741489045Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:28.741965717Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:28.948444515Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:04:30.859508517Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:31.622520983Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:31.623129316Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:31.769655782Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:31.769952385Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:31.957521749Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:31.957952558Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:32.358167904Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:32.358566938Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:32.592567061Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:32.59300918Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:32.905509019Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:33.22309246Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:33.223932337Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:33.797265302Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:33.798956711Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:33.841056915Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:33.843197834Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:33.847426567Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:34.141022425Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:34.141450075Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:35.31047664Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:35.586479288Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:35.586959699Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:36.611086779Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:36.611486512Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:36.922468585Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:37.160722459Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:37.161150723Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:39.371571846Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:39.372284673Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:40.685747188Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:40.686006877Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:04:40.859419595Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:41.245528468Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:41.245866039Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:41.48689297Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:41.52402394Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:41.524393863Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:41.564366169Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:41.564647529Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:41.742804289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:42.544308586Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:42.544803656Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:43.860844127Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:44.003812849Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:44.338033459Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:45.122387006Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:45.122744758Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:45.358117725Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:45.358870482Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:45.722412252Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:45.723154197Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:45.91486264Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:45.915492674Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:46.373755688Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:46.614999904Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:46.615321036Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:46.98574933Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:47.074462074Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:47.074793601Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:48.257323578Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:48.521743648Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:48.522610105Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:48.557423826Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:49.199396841Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:49.557055499Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:49.557289013Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:49.734357197Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:04:50.859453032Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:51.702229181Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:51.702925959Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:51.904714414Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:51.905253662Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:51.986609203Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:52.0672069Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:52.067656346Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:52.577093881Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:52.692528494Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:52.999679792Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:53.000281467Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:53.332469045Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:53.901438896Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:54.973227786Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:54.973445934Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:55.342288926Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:55.342770466Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:55.435758903Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:55.435930345Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:55.743678701Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:55.744085796Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:56.40202232Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:58.27062433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:58.854445131Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:59.296225628Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:59.2964844Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:59.712567555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:59.71288681Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:59.886683529Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:04:59.88684347Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:00.851943225Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:00.852395372Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:05:00.859431441Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:00.930720802Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:01.799229373Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:02.422954074Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:02.423148537Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:03.542922855Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:03.543616103Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:03.699055106Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:04.680337685Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:04.720041345Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:07.178403602Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:07.207956383Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:07.226091728Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:07.226321885Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:07.299755308Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:07.30055569Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:08.015851835Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:08.819814478Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:08.853922647Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:08.854368845Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:09.715944556Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:09.717015632Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:10.031584851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:10.441083624Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:05:10.859431952Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:12.437956831Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:12.438562809Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:12.596246028Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:14.844573085Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:16.714708438Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:16.715158144Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:17.076587197Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:17.39246963Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:17.392784593Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:18.304037308Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:19.410274859Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:19.694916952Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:19.695563239Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:05:20.859428039Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:21.210227252Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:21.21052915Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:21.935509829Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:21.935700326Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:22.760474275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:23.134490981Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:23.134800356Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:25.698346166Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:25.743866155Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:25.744095323Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:26.271881018Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:26.272177612Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:27.384620247Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:27.384922891Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:28.164040531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:28.164631691Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:28.739248046Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:28.739753382Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:28.949165897Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:05:30.859431759Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:31.769492783Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:31.769788408Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:31.958646328Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:31.959099618Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:32.478143771Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:32.478587024Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:32.592698641Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:32.593184947Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:32.676714508Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:32.677447111Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:32.905539764Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:33.422446977Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:33.422769132Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:33.795391482Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:33.796131023Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:33.821235212Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:33.823013991Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:33.826396706Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:34.133445369Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:34.13368631Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:35.310226404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:35.576874158Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:35.577333871Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:36.720294305Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:36.720770977Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:36.963758139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:37.160863284Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:37.161299771Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:39.307631126Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:39.316765204Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:40.688327727Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:40.68862935Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:05:40.859506703Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:41.24859611Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:41.248928726Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:41.487271129Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:41.51788181Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:41.518212076Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:41.563393852Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:41.563655238Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:41.729901702Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:42.558288834Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:42.558754915Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:43.864608808Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:43.998444129Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:44.339121933Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:45.1469411Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:45.147255802Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:45.358179825Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:45.359068474Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:45.713938162Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:45.714607197Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:45.921513976Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:45.922147994Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:46.378051968Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:46.596379791Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:46.596693549Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:46.986711485Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:47.067868428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:47.068278956Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:48.278335975Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:48.553160905Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:48.645995782Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:48.646492245Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:49.205788952Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:49.557162244Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:49.557406338Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:49.659287453Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:05:50.862766672Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:51.704020701Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:51.704859897Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:51.903461558Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:51.903992326Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:51.985835264Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:52.110781063Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:52.111275503Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:52.57718305Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:52.691227369Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:52.998886709Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:52.999306685Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:53.330212224Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:53.896607312Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:54.974381261Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:54.974626807Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:55.33262396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:55.333113657Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:55.475848828Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:55.4761356Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:55.630832806Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:55.631247164Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:56.402575449Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:58.256120409Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:58.852393969Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:59.287296453Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:59.287886661Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:59.71358632Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:59.71390486Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:59.888988552Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:05:59.88914135Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:06:00.85943787Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:00.931333531Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:01.203134373Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:01.203611776Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:02.370701701Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:02.37088468Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:02.67810684Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:02.682682868Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:02.691783615Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:03.697244429Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:04.684737084Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:04.720653503Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:07.195362744Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:07.22615047Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:07.24446625Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:07.244701523Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:07.457731896Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:07.458495328Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:08.013086501Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:08.777903425Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:08.850204829Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:08.850811095Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:09.691296789Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:09.691809269Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:10.010928164Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:10.441451477Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:06:10.871735061Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:12.442399715Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:12.442995481Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:12.517494309Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:14.846833651Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:16.717485404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:16.717944909Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:17.072401716Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:17.408482678Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:17.408877585Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:18.230383717Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:19.404941639Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:19.992223958Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:19.993045262Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:06:20.859431864Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:21.208969851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:21.209359188Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:21.923155523Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:21.923332409Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:22.773063885Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:23.139263963Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:23.139542279Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:25.698378695Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:25.744244252Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:25.744472171Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:26.348516925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:26.348863997Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:27.378812067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:27.379100108Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:28.181637752Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:28.182240459Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:28.583149451Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:28.583712164Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:28.945238867Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:06:30.859452658Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:31.769310656Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:31.769586488Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:32.274842079Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:32.275242529Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:32.450083448Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:32.45047661Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:32.519510223Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:32.520020937Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:32.593757245Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:32.594172703Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:32.905869479Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:33.213728903Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:33.214073274Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:33.795283714Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:33.796070783Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:33.815949137Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:33.817167842Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:33.819454957Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:34.132750606Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:34.132980275Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:35.335396921Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:35.58662752Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:35.587478216Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:36.632349721Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:36.632759946Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:36.964746997Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:37.160843484Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:37.161357348Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:39.171993421Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:39.172359753Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:40.706218752Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:40.706495815Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:06:40.859479533Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:41.064781834Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:41.065069115Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:41.486969143Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:41.516738355Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:41.517085735Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:41.566315722Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:41.566613131Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:41.716001515Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:42.546951319Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:42.547361435Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:43.866544823Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:43.998152692Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:44.338100898Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:45.120022971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:45.120307992Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:45.358126775Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:45.358848013Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:45.738178226Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:45.739253814Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:45.939809771Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:45.940629606Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:46.375187716Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:46.768938239Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:46.769386523Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:46.985425459Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:47.069811969Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:47.070212578Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:48.268846497Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:48.55463489Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:48.646001284Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:48.646508267Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:49.201582077Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:49.557370141Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:49.557610003Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:49.642862411Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:06:50.859439325Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:51.717771551Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:51.718485123Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:51.91556735Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:51.916235227Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:51.983717221Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:52.070164037Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:52.070588339Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:52.586106648Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:52.691210336Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:52.998963255Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:52.999411913Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:53.331722186Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:53.896725701Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:54.978241232Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:54.978471654Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:55.296627807Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:55.297072258Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:55.413173671Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:55.413322734Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:55.645716242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:55.646133101Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:56.41175107Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:58.260530082Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:58.852206341Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:59.101643487Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:59.101935447Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:59.722106383Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:59.722387423Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:59.889845365Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:06:59.889989324Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:07:00.859463312Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:00.930209346Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:01.054190056Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:01.054690376Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:01.675939135Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:02.403997842Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:02.40419672Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:02.731525357Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:02.732438861Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:03.697837804Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:04.676045888Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:04.720550424Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:07.180242998Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:07.210229749Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:07.229398988Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:07.229650019Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:07.302497128Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:07.303251949Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:08.013929606Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:08.196829363Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:09.515720582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:09.516146169Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:09.722317143Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:09.73048322Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:10.020717593Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:10.430373109Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:07:10.859431798Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:12.440581213Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:12.441204472Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:12.595939699Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:14.840475771Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:16.844064965Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:16.844612124Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:17.076206744Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:17.433274454Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:17.43357769Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:18.164155455Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:19.406137157Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:19.702292011Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:19.70282422Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:07:20.859511571Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:21.207676832Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:21.208028587Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:21.908171033Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:21.908362062Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:22.760885582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:23.134984276Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:23.135226192Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:25.69794359Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:25.743933221Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:25.744164392Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:26.272382392Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:26.272689593Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:27.312438111Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:27.312791098Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:28.166114469Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:28.166797893Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:28.599233967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:28.599875803Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:28.937972852Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:07:30.859426455Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:31.76945581Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:31.769719212Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:32.285262685Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:32.285646704Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:32.455579277Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:32.45596769Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:32.548230056Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:32.548797275Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:32.59235486Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:32.592820636Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:32.905593456Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:33.213470937Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:33.213816736Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:33.795280835Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:33.79603265Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:33.815763064Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:33.816919938Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:33.819147582Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:34.133562383Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:34.133835907Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:35.310428711Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:35.576754566Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:35.577175035Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:36.620107187Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:36.620628153Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:36.920859627Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:37.160601661Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:37.161012404Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:39.138992475Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:39.139352258Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:40.743489621Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:40.743963636Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:07:40.860757696Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:41.368485417Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:41.369563253Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:41.496318408Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:41.631134117Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:41.631574659Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:41.633387835Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:41.633963103Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:41.699340448Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:42.548886516Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:42.549312305Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:43.859103155Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:43.997922599Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:44.279333833Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:44.556092423Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:44.556380408Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:45.358339504Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:45.359141002Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:45.748624027Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:45.749277341Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:45.91474396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:45.91541945Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:46.377580148Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:46.605205228Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:46.605620434Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:46.986290076Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:47.073378854Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:47.073681485Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:48.258379839Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:48.554085329Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:48.643911365Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:48.644616081Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:49.201108182Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:49.55544914Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:49.555717865Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:49.648573003Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:07:50.859456817Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:51.711079337Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:51.711784592Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:51.899344355Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:51.899905625Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:51.942607487Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:52.077630376Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:52.078050217Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:52.576695179Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:52.692406212Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:52.998749645Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:52.999213331Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:53.330055875Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:53.89610601Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:54.972261097Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:54.972472665Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:55.296319935Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:55.296787808Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:55.431038793Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:55.431207884Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:55.75752535Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:55.757969097Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:56.40510018Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:58.265870278Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:58.86030029Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:59.098568797Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:59.098885064Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:59.710135445Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:59.710412514Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:59.909410409Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:07:59.909564367Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:08:00.859436254Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:00.930309696Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:01.003063895Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:01.003555942Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:01.790413257Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:02.378990235Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:02.379134909Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:02.673595869Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:02.67425812Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:03.702104115Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:04.676845844Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:04.720507127Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:07.178900906Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:07.208834612Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:07.233942091Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:07.23421898Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:07.27390223Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:07.274596144Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:08.011562541Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:08.938003887Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:09.535952054Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:09.566350753Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:09.702307325Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:09.703225522Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:10.034025651Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:10.431224375Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:08:10.859439512Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:12.431059415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:12.431586508Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:12.596911366Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:14.844577547Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:16.842254034Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:16.842800313Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:17.098532127Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:17.434319907Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:17.434636163Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:18.285543958Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:19.408627309Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:19.694086483Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:19.694585537Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:08:20.859428302Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:21.261728901Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:21.262077075Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:21.901309396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:21.901468392Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:22.775159108Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:23.140650131Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:23.140944624Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:25.719074421Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:25.744077275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:25.744310582Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:26.293894848Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:26.294180023Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:27.381332187Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:27.381598114Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:28.189092853Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:28.190135644Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:28.622242206Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:28.623392827Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:28.973613306Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:08:30.859493984Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:31.76923545Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:31.769467264Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:32.298708405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:32.299093907Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:32.49848801Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:32.498873375Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:32.545132435Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:32.545652208Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:32.59250752Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:32.592974132Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:32.905516932Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:33.218753Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:33.2191009Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:33.795414458Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:33.796181581Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:33.815830923Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:33.816965599Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:33.819087025Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:34.132587598Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:34.132851035Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:35.310910052Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:35.586364036Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:35.586914384Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:36.720336035Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:36.721332642Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:36.966517922Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:37.183294724Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:37.186335638Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:39.291366099Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:39.29202936Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:40.719593369Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:40.719985371Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:08:40.859484081Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:41.258725093Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:41.259064081Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:41.49429595Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:41.498941677Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:41.499379856Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:41.564472093Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:41.564823307Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:41.713077116Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:42.560110066Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:42.560528586Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:43.859090474Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:43.997577175Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:44.337750104Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:45.124354026Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:45.124695334Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:45.358197459Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:45.358954061Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:45.716195227Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:45.716887749Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:45.91336742Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:45.913991472Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:46.377784734Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:46.601826036Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:46.602162499Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:46.985482516Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:47.094373106Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:47.094782544Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:48.259898681Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:48.556260478Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:48.650010528Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:48.650586965Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:49.202190649Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:49.503917913Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:49.504199823Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:49.66096925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:08:50.859440553Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:51.700926107Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:51.701612599Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:51.911417053Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:51.911993835Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:51.984980868Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:52.075256353Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:52.075678983Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:52.577004041Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:52.691368504Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:52.998717624Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:52.99918721Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:53.332127017Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:53.896967664Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:55.003261448Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:55.003487229Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:55.295889494Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:55.296382562Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:55.412953054Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:55.413102217Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:55.763595867Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:55.764082711Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:56.405968664Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:58.258784836Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:58.853254611Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:59.286722201Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:59.287008019Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:59.711791519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:59.712053059Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:59.90614183Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:08:59.9062716Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:09:00.859430548Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:00.930966949Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:01.126146601Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:01.126646965Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:01.72628764Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:02.34552402Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:02.345708788Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:02.686481745Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:02.701469406Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:03.699982753Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:04.676308802Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:04.720298278Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:07.181531428Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:07.225714351Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:07.251876282Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:07.252149696Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:07.279493295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:07.280183088Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:08.01598106Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:08.125248276Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:09.464538019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:09.465362619Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:09.681771488Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:09.682393162Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:10.083006475Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:10.435210318Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:09:10.859427983Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:12.485779728Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:12.486419961Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:12.530596903Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:15.118708555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:16.714372028Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:16.714860379Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:17.073960676Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:17.518133823Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:17.518478362Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:18.459552722Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:19.406174537Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:19.68640339Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:19.686916657Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:09:20.859441941Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:21.221618659Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:21.221971569Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:21.910223503Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:21.910410583Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:22.758537379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:23.134423796Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:23.134704135Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:25.697602663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:25.744137253Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:25.744352295Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:26.271991063Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:26.272299562Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:28.165956069Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:28.166531371Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:28.806312191Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:28.806836026Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:28.936625107Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:09:30.85942943Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:31.76902433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:31.769291006Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:32.273349452Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:32.273762746Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:32.349935171Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:32.350264174Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:32.497331678Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:32.497927786Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:32.583232606Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:32.583718815Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:32.905907497Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:33.214031114Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:33.214364294Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:33.795365033Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:33.796269562Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:33.823032024Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:33.824884975Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:33.827227546Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:34.132699311Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:34.132946761Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:35.310499978Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:35.576905864Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:35.577344553Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:36.697300754Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:36.697714625Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:36.925166104Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:37.160642834Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:37.161097481Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:37.302791227Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:37.303052755Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:39.098290587Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:39.098762008Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:40.6932357Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:40.693493864Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:09:40.867740624Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:41.277806563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:41.278218753Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:41.489541193Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:41.511523894Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:41.511963905Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:41.586855338Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:41.587163961Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:41.72223667Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:42.226259096Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:42.226738824Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:43.855864433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:43.997143458Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:44.272099596Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:45.135854972Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:45.136162331Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:45.358082251Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:45.358879057Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:45.725053698Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:45.725718525Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:45.91617538Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:45.916808737Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:46.376431557Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:46.603738271Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:46.604192124Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:46.982783386Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:47.077910529Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:47.078216594Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:48.257741829Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:48.553125957Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:48.648972589Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:48.649548844Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:49.204126794Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:49.493106239Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:49.493342911Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:49.6563166Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:09:50.859411677Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:51.710944414Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:51.711750997Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:51.912077329Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:51.91261151Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:51.97838092Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:52.074356181Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:52.074792423Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:52.578648979Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:52.697256181Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:53.024572798Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:53.025244785Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:53.346067283Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:53.898815912Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:54.973645397Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:54.973893169Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:55.346996934Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:55.34747456Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:55.41392448Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:55.414060312Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:55.812419183Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:55.812860768Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:56.403371295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:58.294882942Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:58.869379834Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:59.289328951Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:59.289645283Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:59.712106731Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:59.712431261Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:59.899600493Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:09:59.899767047Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:10:00.859429178Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:00.930137691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:01.000415422Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:01.000896611Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:01.721022354Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:02.617274563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:02.657917833Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:04.306232808Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:04.34222187Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:04.343320599Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:04.74215239Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:04.77662161Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:07.238736494Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:07.273305487Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:07.298128965Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:07.298380468Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:07.299913109Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:07.300732201Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:08.012488143Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:08.488737889Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:09.427236424Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:09.428947277Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:09.698622268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:09.699305669Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:10.074705445Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:10.446077088Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:10:10.867813099Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:12.431132242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:12.43176755Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:12.595151641Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:15.032775902Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:16.85777512Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:16.858195851Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:17.075776838Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:17.39963284Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:17.400000268Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:18.184808905Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:19.408077155Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:19.717479851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:19.718207078Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:10:20.859502554Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:21.21327585Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:21.213645124Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:21.965992099Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:21.966173637Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:22.77048526Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:23.135093006Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:23.135388285Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:25.698067119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:25.866547448Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:25.86696678Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:26.27183591Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:26.272105734Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:27.388039221Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:27.388335167Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:28.177174205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:28.177603234Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:28.716403543Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:28.716975375Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:28.962040777Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:10:30.859435458Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:31.652222314Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:31.652798966Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:31.776373851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:31.776614646Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:31.964686347Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:31.965228318Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:32.396318822Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:32.396794344Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:32.620891045Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:32.621503081Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:32.911233647Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:33.214609324Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:33.215196969Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:33.795375923Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:33.796167318Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:33.816884308Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:33.818050443Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:33.820252788Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:34.132495524Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:34.13277757Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:35.311616582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:35.577391911Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:35.577996925Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:36.613147707Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:36.613599113Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:36.921527745Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:37.161747542Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:37.162128198Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:39.688896799Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:39.689933548Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:40.802141783Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:40.802627955Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:10:40.864287015Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:41.261267448Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:41.262104323Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:41.493393687Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:41.535576494Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:41.537743454Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:41.569418712Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:41.570161138Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:41.72175458Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:42.557523773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:42.558038715Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:43.857781168Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:44.000759082Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:44.338554096Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:45.17809993Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:45.178359564Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:45.358087465Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:45.358845991Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:45.733264468Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:45.733984568Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:45.958205893Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:45.958816306Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:46.381039892Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:46.605697986Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:46.606040653Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:46.981787732Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:47.225358726Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:47.225692743Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:48.260428955Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:48.680997249Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:48.681592813Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:48.682117574Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:49.228482935Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:49.606047702Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:49.606551408Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:49.658713736Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:10:50.859437248Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:51.710339617Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:51.711294532Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:51.912904457Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:51.913421083Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:51.987718848Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:52.068353596Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:52.068769664Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:52.578591877Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:52.699292017Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:52.999976879Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:53.000563299Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:53.336522401Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:53.902193688Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:54.984763415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:54.984989122Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:55.433306864Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:55.433440565Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:55.467975269Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:55.468459685Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:55.79707097Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:55.797543644Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:56.402933354Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:58.314067923Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:58.855144488Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:59.117309161Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:59.117659279Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:59.715600555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:59.715891091Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:59.870729578Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:10:59.870882306Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:11:00.859433351Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:00.931571392Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:01.026093099Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:01.026581356Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:01.61570982Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:02.34568993Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:02.345851669Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:02.678473121Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:02.679306419Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:03.697932313Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:04.690336689Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:04.722975039Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:07.182366181Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:07.214575216Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:07.24080076Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:07.241119938Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:07.275163463Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:07.275894345Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:08.017581755Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:08.866883586Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:09.561356053Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:09.562342114Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:09.695107503Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:09.697001508Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:10.017910141Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:10.432550608Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:11:10.859427622Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:12.438282554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:12.438898246Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:12.596591738Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:14.841023613Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:16.842371708Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:16.842827065Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:17.07662444Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:17.433642571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:17.434006723Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:18.165000345Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:19.410050116Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:19.6820619Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:19.682578557Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:11:20.859439847Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:21.230216471Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:21.230535337Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:21.922238979Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:21.922393431Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:22.759336524Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:23.13991983Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:23.140261161Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:25.698601798Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:25.744311799Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:25.744580569Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:26.272167436Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:26.272460525Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:27.37824906Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:27.378508886Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:28.160612554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:28.161637367Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:28.788862909Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:28.789311549Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:28.937976642Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:11:30.859422616Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:31.691723464Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:31.692256005Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:31.769184875Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:31.769377581Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:31.973370091Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:31.97376547Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:32.346814027Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:32.347149409Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:32.593246857Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:32.593753639Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:32.905658917Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:33.213628232Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:33.213986868Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:33.795528567Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:33.796293806Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:33.816787266Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:33.817956162Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:33.820210678Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:34.132502657Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:34.132744268Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:35.311937039Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:35.577833989Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:35.578447085Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:36.614038308Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:36.614452568Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:36.926272747Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:37.160947389Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:37.161376972Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:39.265883049Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:39.266371745Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:40.715528036Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:40.715799503Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:11:40.859509347Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:41.251187246Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:41.251518422Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:41.429900925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:41.430224316Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:41.48683806Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:41.561162624Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:41.561599093Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:41.735775516Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:42.541528969Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:42.541973692Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:43.859567193Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:43.998033522Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:44.33848911Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:45.126517182Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:45.126890055Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:45.358096834Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:45.358963422Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:45.727915275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:45.728817477Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:45.916533164Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:45.917316895Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:46.380565982Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:46.632854283Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:46.633150628Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:46.982600225Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:47.227302373Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:47.227564425Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:48.255796779Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:48.55440971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:48.648293302Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:48.648904321Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:49.201013355Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:49.557617051Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:49.557870103Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:49.657922996Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:11:50.861069404Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:51.708448344Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:51.709168068Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:51.900336158Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:51.901015763Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:51.984777117Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:52.074192262Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:52.074618473Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:52.578096489Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:52.693392802Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:52.998758597Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:52.999179989Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:53.301032881Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:53.896801837Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:54.972740253Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:54.972959801Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:55.333045997Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:55.3334635Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:55.412458816Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:55.41258265Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:55.642234507Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:55.642693086Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:56.4022294Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:58.282323282Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:58.856518489Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:59.277199053Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:59.277504976Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:59.711634566Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:59.71196Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:59.889657442Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:11:59.889831566Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:12:00.859442267Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:00.930150615Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:01.098205413Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:01.098763163Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:01.716681183Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:02.349514372Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:02.349644129Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:02.679286442Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:02.680176063Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:03.697966975Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:04.770651004Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:04.853460875Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:07.189743218Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:07.221783656Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:07.244037366Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:07.244253457Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:07.272057244Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:07.272815651Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:08.012221495Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:08.226006757Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:09.396326011Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:09.396608441Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:09.699404769Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:09.699920181Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:10.011965043Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:10.48296418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:12:10.884737542Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:12.431955566Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:12.432565678Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:12.517354456Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:14.860792803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:16.724614788Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:16.725112996Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:17.043795875Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:17.428160708Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:17.42846566Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:18.15701732Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:19.407560771Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:19.694303129Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:19.694871113Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:12:20.859415739Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:21.210050407Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:21.210407293Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:21.909650214Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:21.909842851Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:22.774133086Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:23.150967282Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:23.151297174Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:25.698483522Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:25.744015527Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:25.744265668Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:26.272076653Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:26.272374474Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:27.381549742Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:27.381850338Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:28.160014284Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:28.160588334Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:28.729140095Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:28.729540892Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:28.95406071Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:12:30.859430225Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:31.6424425Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:31.643238599Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:31.769771326Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:31.770023838Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:31.96643088Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:31.966841763Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:32.518477952Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:32.51887414Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:32.600347555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:32.600911617Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:32.905780814Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:33.213482173Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:33.213830268Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:33.79530234Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:33.796274368Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:33.817052039Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:33.818227457Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:33.820429007Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:34.133045945Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:34.13328781Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:35.312466888Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:35.590023367Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:35.590609172Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:36.653020428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:36.653436618Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:36.920540115Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:37.161279814Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:37.161767184Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:39.311935657Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:39.316649275Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:40.701168905Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:40.701417467Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:12:40.859441742Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:41.26033446Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:41.260923825Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:41.489945146Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:41.516074588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:41.516457009Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:41.560157626Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:41.560488313Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:41.718573997Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:42.242821988Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:42.243969769Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:43.860423886Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:43.997204452Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:44.33832387Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:44.551122875Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:44.551408913Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:45.358114814Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:45.358854337Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:45.723715672Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:45.724597979Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:45.929641891Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:45.930282638Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:46.377395001Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:46.611863309Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:46.612246604Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:46.983115896Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:47.214495374Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:47.214835906Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:48.257590495Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:48.517619319Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:48.518246641Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:48.55754324Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:49.201487153Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:49.555581198Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:49.555830233Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:49.657345477Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:12:50.859428939Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:51.916310508Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:51.917010335Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:51.967174726Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:51.967883924Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:51.986435999Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:52.085592921Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:52.086276713Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:52.577766858Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:52.691728752Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:52.998968973Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:52.99941364Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:53.315061467Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:53.896829232Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:54.972618221Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:54.972851804Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:55.340973173Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:55.341429439Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:55.432984924Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:55.433127355Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:55.770784166Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:55.771300096Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:56.40302915Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:58.305376577Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:58.853992007Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:59.279449661Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:59.27977995Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:59.723184869Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:59.723487826Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:59.872527244Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:12:59.872827134Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:13:00.859495456Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:00.93021358Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:01.001254373Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:01.001795064Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:01.539211171Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:02.379269941Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:02.379422694Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:02.708752415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:02.709440639Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:03.698914616Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:04.678267538Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:04.719940614Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:07.222387838Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:07.268712173Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:07.294055191Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:07.294293142Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:07.324708551Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:07.32578377Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:08.012963504Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:08.351722541Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:09.17085738Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:09.171282487Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:09.691450427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:09.691933641Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:10.014529629Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:10.434803397Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:13:10.85982166Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:12.440915353Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:12.441463947Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:12.515917264Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:14.845822454Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:16.636602604Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:16.637040817Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:16.714499646Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:16.714969567Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:17.078442144Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:18.288049151Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:19.408658565Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:19.992000284Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:19.992640424Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:13:20.859409036Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:21.243088698Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:21.243540396Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:21.949414401Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:21.949589852Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:22.772821798Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:23.137030576Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:23.137299826Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:25.697824602Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:25.743814863Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:25.744033767Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:26.340872632Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:26.341235438Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:27.381301172Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:27.381563513Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:28.159122202Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:28.159557283Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:28.567906355Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:28.568429951Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:28.936348766Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:13:30.859491865Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:31.641387821Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:31.642074807Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:31.769560833Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:31.76983346Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:32.280071838Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:32.280488795Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:32.459018028Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:32.459344004Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:32.565480446Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:32.565957545Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:32.905787556Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:33.214016846Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:33.214329468Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:33.795363971Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:33.796119058Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:33.816088869Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:33.817219869Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:33.819398849Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:34.132479775Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:34.132718587Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:35.317273284Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:35.576994912Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:35.577448726Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:36.606983211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:36.608027653Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:36.926100119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:37.16829649Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:37.168756864Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:39.175782769Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:39.176639771Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:40.688631211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:40.688973346Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:13:40.859444751Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:41.065679426Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:41.066118474Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:41.486761288Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:41.523084Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:41.52346911Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:41.55733081Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:41.557733267Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:41.750204177Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:42.547404787Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:42.547902288Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:43.858540844Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:43.998702478Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:44.338184829Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:45.121210701Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:45.121521622Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:45.358090694Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:45.358886762Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:45.725789418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:45.726452249Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:45.922391548Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:45.923015443Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:46.375633289Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:46.590572663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:46.590918413Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:46.984792328Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:47.105761719Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:47.106113842Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:48.254175074Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:48.53757416Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:48.541221071Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:48.553839409Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:49.202351833Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:49.562681657Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:49.562975529Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:49.655354849Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:13:50.859435857Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:51.712477905Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:51.713184914Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:51.91133726Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:51.912068015Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:51.982866271Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:52.077166908Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:52.077539619Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:52.577969026Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:52.692389776Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:53.002944941Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:53.00337653Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:53.311645356Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:53.897094688Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:54.972232033Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:54.972453173Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:55.29637708Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:55.296833101Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:55.413407477Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:55.413564967Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:55.75314648Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:55.753593447Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:56.420120253Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:58.314501424Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:58.858428924Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:59.308124192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:59.308513558Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:59.722314316Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:13:59.722710665Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:00.038697811Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:00.038858223Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:14:00.859514215Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:00.932235327Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:01.0701881Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:01.070647673Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:01.594680953Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:02.379006568Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:02.379166047Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:02.671222064Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:02.671927695Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:03.70083581Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:04.678224009Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:04.720579195Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:07.183818418Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:07.228969103Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:07.252227049Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:07.252540254Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:07.337732103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:07.338587829Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:08.016496349Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:08.406431357Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:08.806993591Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:08.807380586Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:09.691194254Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:09.691630493Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:10.012763498Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:10.431965173Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:14:10.859524677Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:12.480746498Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:12.481534303Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:12.524638667Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:14.849932688Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:16.855308752Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:16.855757038Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:17.07643698Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:17.490344164Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:17.490789097Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:18.208724321Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:19.406263718Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:19.697446328Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:19.698023015Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:14:20.859480849Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:21.207542788Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:21.207896232Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:21.928814729Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:21.92899104Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:22.773128213Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:23.136997969Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:23.137323448Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:25.698994802Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:25.74402658Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:25.74424981Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:26.272174938Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:26.272454154Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:27.311253944Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:27.311503953Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:28.160302455Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:28.161752916Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:28.727828647Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:28.728193223Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:28.946379323Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:14:30.85941759Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:31.630062914Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:31.63071523Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:31.769491499Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:31.769785443Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:32.272277902Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:32.272702617Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:32.452554012Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:32.452930528Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:32.596145679Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:32.596633604Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:32.905674182Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:33.221403023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:33.221739193Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:33.795278751Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:33.796071082Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:33.815536568Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:33.816880507Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:33.819024472Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:34.178077135Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:34.17833652Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:35.310616113Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:35.57679645Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:35.577234669Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:36.595431607Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:36.59584429Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:36.921198868Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:37.161048544Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:37.161502611Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:39.227229518Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:39.229338239Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:40.700803304Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:40.701075166Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:14:40.85942896Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:41.263135582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:41.263453042Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:41.487682442Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:41.549822229Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:41.55020511Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:41.556646303Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:41.556914588Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:41.723639697Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:42.550688395Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:42.551093555Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:43.882692947Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:44.002233101Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:44.338517978Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:45.121045822Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:45.121363266Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:45.358155438Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:45.358938857Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:45.724364298Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:45.725092121Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:45.926174705Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:45.926792108Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:46.46575188Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:46.612334121Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:46.612702988Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:46.985027932Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:47.076216539Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:47.076503596Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:48.289338363Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:48.513225095Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:48.514043455Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:48.554011225Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:49.200431269Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:49.503778534Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:49.504020639Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:49.64445538Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:14:50.859432785Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:51.703869163Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:51.704574167Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:51.891586551Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:51.892138484Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:51.982156452Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:52.068475862Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:52.068907521Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:52.577057248Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:52.691588394Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:52.999175678Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:52.999605508Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:53.300840563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:53.897072294Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:54.99423744Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:54.994433841Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:55.306918556Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:55.307504241Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:55.476931767Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:55.477107731Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:55.756335261Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:55.756817272Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:56.402655225Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:58.295102317Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:58.852110198Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:59.284882491Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:59.285224116Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:59.714713969Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:59.714992309Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:59.899230089Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:14:59.899405251Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:15:00.859510173Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:00.93024871Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:01.064025161Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:01.06452611Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:02.693523285Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:02.776868066Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:03.405578682Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:03.469956137Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:03.471068927Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:03.698051457Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:04.679621784Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:04.7214307Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:07.204638583Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:07.240750428Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:07.261072133Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:07.261330748Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:07.279786119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:07.280567488Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:08.011778291Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:09.465637991Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:09.467298901Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:09.618379612Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:09.753268383Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:09.758823553Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:10.05564189Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:10.440982524Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:15:10.859374855Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:12.448677832Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:12.450048981Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:12.519652615Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:14.870863747Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:16.842743456Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:16.843176919Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:17.074566442Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:17.424652595Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:17.425003987Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:18.27491065Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:19.408405565Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:19.737162514Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:19.737869481Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:15:20.859427267Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:21.210310457Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:21.210707768Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:21.923266937Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:21.923428596Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:22.772516174Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:23.159162483Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:23.159418459Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:25.697869604Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:25.743965306Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:25.744205004Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:26.299066323Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:26.299351299Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:28.178752756Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:28.179365491Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:28.609577646Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:28.610147113Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:28.942041283Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:29.811217082Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:29.811512289Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:15:30.859433822Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:31.661470606Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:31.662248305Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:31.769063878Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:31.769275793Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:32.29440095Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:32.294794006Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:32.486651775Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:32.487028162Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:32.596843603Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:32.597270911Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:32.905565634Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:33.21615806Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:33.216492725Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:33.795517458Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:33.796240407Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:33.815439591Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:33.816558012Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:33.818792287Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:34.133013381Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:34.133311341Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:35.31056257Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:35.576771024Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:35.577230122Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:36.614703197Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:36.615205437Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:36.924532623Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:37.161242387Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:37.161700253Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:39.34886119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:39.349482334Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:40.700911572Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:40.701168539Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:15:40.859434043Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:41.263266408Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:41.263581791Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:41.487297703Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:41.516253503Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:41.516572638Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:41.558418889Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:41.558814905Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:41.710088394Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:42.566970088Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:42.567428152Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:43.860740915Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:44.001568581Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:44.272275163Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:45.135482817Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:45.135825989Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:45.358047557Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:45.358806363Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:45.73451736Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:45.73520792Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:45.954424559Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:45.955048774Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:46.376614718Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:46.76096779Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:46.761333719Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:46.982894194Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:47.083706072Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:47.084007646Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:48.263394294Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:48.553943687Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:48.654181957Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:48.655001037Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:49.205503488Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:49.504322779Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:49.504584903Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:49.652765436Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:15:50.859507652Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:51.712207421Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:51.713138276Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:51.911231253Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:51.912055951Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:51.979013587Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:52.097481414Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:52.097899575Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:52.577375792Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:52.691086788Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:52.998713041Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:52.999147684Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:53.324941417Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:53.896538291Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:54.972520026Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:54.972771467Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:55.296233033Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:55.296700706Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:55.507030796Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:55.507168922Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:55.751861645Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:55.752298479Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:56.403061721Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:58.296619386Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:58.851993275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:59.292561984Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:59.292960634Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:59.718504721Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:59.718760922Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:59.88640571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:15:59.886566264Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:16:00.859432092Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:00.93204902Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:01.04494273Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:01.045486209Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:01.584072674Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:02.369821466Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:02.369996462Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:03.136375791Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:03.137212092Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:03.700103314Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:04.722945624Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:04.793524147Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:07.182495554Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:07.23104976Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:07.253687968Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:07.253922221Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:07.291685286Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:07.292415688Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:08.043490171Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:08.52886146Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:09.609142937Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:09.609529884Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:09.732433101Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:09.733159546Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:10.048532972Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:10.432213874Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:16:10.859527401Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:12.459186715Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:12.459799163Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:12.520825713Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:14.840528453Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:16.836086784Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:16.836507291Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:17.072219776Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:17.392643298Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:17.392941549Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:18.208745104Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:19.406870851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:20.000600193Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:20.001193289Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:16:20.859428617Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:21.244839446Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:21.24529744Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:21.919030049Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:21.919180669Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:22.773235336Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:23.134562044Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:23.134841301Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:25.698184062Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:25.744308468Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:25.744535879Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:26.342473947Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:26.342781551Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:27.381273487Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:27.38153857Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:28.163093113Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:28.164199996Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:28.579806981Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:28.580338468Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:28.939356697Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:16:30.859495289Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:31.768949861Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:31.769214018Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:32.030926849Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:32.031498499Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:32.286517488Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:32.286926269Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:32.363356022Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:32.363725105Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:32.593024187Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:32.593463296Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:32.905561243Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:33.223279834Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:33.223762339Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:33.79530543Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:33.796062667Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:33.81743344Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:33.81859099Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:33.820778115Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:34.132623084Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:34.132899331Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:35.314938457Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:35.577170152Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:35.57764673Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:36.619189114Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:36.619568553Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:36.921335172Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:37.16109181Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:37.161527608Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:39.274426279Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:39.276601401Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:40.701003334Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:40.701277713Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:16:40.859429859Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:41.262585461Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:41.262982021Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:41.487305219Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:41.512224365Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:41.51257951Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:41.596427272Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:41.596855169Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:41.730839458Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:42.556468852Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:42.556961921Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:43.864429333Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:43.997926674Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:44.338029117Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:45.134803407Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:45.135604677Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:45.365429141Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:45.367223268Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:45.749028493Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:45.750648417Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:45.939613861Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:45.940957354Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:46.37463353Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:46.771881295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:46.772342225Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:46.980593254Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:47.075140379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:47.075431711Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:48.257899839Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:48.562581812Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:48.647596783Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:48.648290153Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:49.20140676Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:49.557353229Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:49.558415985Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:49.653550645Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:16:50.859415899Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:51.777200816Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:51.778081023Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:51.903967124Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:51.904600819Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:51.942584744Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:52.136302355Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:52.136710979Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:52.578009939Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:52.691723526Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:53.004825955Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:53.005208371Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:53.300500898Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:53.897000884Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:54.979198417Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:54.979423077Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:55.296085248Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:55.296539481Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:55.47678381Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:55.476944353Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:55.746047869Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:55.746415117Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:56.409989551Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:58.287271869Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:58.854480751Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:59.098134712Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:59.098459064Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:59.741268223Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:59.741556867Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:59.893303891Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:16:59.893445296Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:17:00.85944056Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:00.930704446Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:00.975042692Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:00.975517443Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:01.514289325Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:02.353540895Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:02.353707307Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:02.69311581Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:02.69376987Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:03.699149094Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:04.687624878Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:04.720967466Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:07.182465982Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:07.214446104Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:07.233384685Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:07.233658146Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:07.276723363Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:07.277434278Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:08.024374636Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:08.051564742Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:09.458415863Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:09.458791189Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:09.699407827Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:09.702881806Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:10.040149439Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:10.433074962Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:17:10.859431394Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:12.439382598Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:12.439975206Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:12.51592128Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:14.843184949Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:16.714781998Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:16.715232973Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:17.074361375Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:17.408267898Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:17.408579741Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:18.176934747Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:19.406375594Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:19.696245997Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:19.696830304Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:17:20.859429116Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:21.223675911Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:21.224006338Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:21.924980724Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:21.92517488Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:22.760142097Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:23.134802444Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:23.135066955Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:25.697125433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:25.745266563Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:25.745511931Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:26.306538889Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:26.30684483Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:27.311150988Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:27.311433712Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:28.159103728Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:28.159911745Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:28.762205086Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:28.76489357Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:28.93130695Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:17:30.859486533Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:31.770070728Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:31.770311802Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:31.955956989Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:31.956335144Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:32.35750777Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:32.357847861Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:32.548608645Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:32.549116616Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:32.592724996Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:32.593169654Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:32.905551414Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:33.219504574Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:33.219891339Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:33.79537305Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:33.796110743Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:33.816276547Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:33.817400117Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:33.81957883Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:34.132656434Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:34.132934459Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:35.312119698Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:35.577006324Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:35.577604409Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:36.616646202Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:36.617147938Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:36.923890345Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:37.16074571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:37.161185468Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:39.188801393Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:39.189313217Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:40.701628196Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:40.701897959Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:17:40.859435187Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:41.262970265Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:41.263295958Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:41.491068526Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:41.511456693Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:41.511829723Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:41.555950004Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:41.556202094Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:41.722293229Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:42.552440497Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:42.552929466Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:43.862111646Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:43.999261866Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:44.282066481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:44.549194413Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:44.549505759Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:45.358116176Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:45.358981613Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:45.751368418Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:45.752031726Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:45.924121993Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:45.924740781Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:46.37720467Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:46.606828162Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:46.60717603Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:46.983152532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:47.080336639Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:47.080644228Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:48.25702094Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:48.554160286Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:48.708742843Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:48.709333432Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:49.212950656Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:49.555235661Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:49.555458115Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:49.640895471Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:17:50.859427801Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:51.700750914Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:51.701433535Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:51.921651642Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:51.922298288Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:51.987262381Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:52.073942259Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:52.074366745Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:52.577142205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:52.690864997Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:52.999907699Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:53.000483326Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:53.301132254Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:53.897010266Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:54.974350256Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:54.974561499Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:55.333425944Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:55.333942172Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:55.413097388Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:55.413249465Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:55.824029846Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:55.824585274Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:56.411001516Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:58.285829193Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:58.852033244Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:59.098722939Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:59.098981648Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:59.725346752Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:59.725584034Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:59.88723223Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:17:59.887390776Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:18:00.859428898Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:00.930425457Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:01.065604662Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:01.066039101Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:02.111294389Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:02.345407192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:02.345551424Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:02.692225295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:02.692934612Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:03.790641033Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:04.675122209Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:04.727008396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:07.179267805Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:07.208896485Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:07.228060308Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:07.228262445Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:07.292062449Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:07.293013657Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:08.015654378Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:08.295813064Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:09.529606791Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:09.529987063Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:09.706655926Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:09.707429292Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:10.146983148Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:10.430387576Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:18:10.859452055Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:12.442180073Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:12.442794037Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:12.597398442Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:14.844215767Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:16.724551209Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:16.72501736Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:17.076505455Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:17.410345297Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:17.410688566Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:18.25010096Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:19.408250362Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:19.719362928Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:19.719951542Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:18:20.859375984Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:21.234090116Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:21.234443535Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:21.935981157Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:21.936165519Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:22.760003426Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:23.134628211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:23.134904804Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:25.697610518Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:25.744982713Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:25.745214072Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:26.300212601Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:26.30051188Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:27.37999258Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:27.380289965Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:28.183030686Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:28.183608372Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:28.671192964Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:28.671735617Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:28.947222321Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:18:30.859433022Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:31.622805898Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:31.62335602Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:31.769230765Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:31.769443992Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:31.957932344Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:31.958410658Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:32.352315184Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:32.352711388Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:32.593200839Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:32.593599212Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:32.905606278Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:33.213801179Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:33.214084356Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:33.820612912Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:33.821914842Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:33.846789691Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:33.848110732Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:33.850471455Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:34.132737583Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:34.133006626Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:35.309896556Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:35.577067514Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:35.577531352Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:36.599153576Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:36.599549067Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:36.922204694Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:37.176753142Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:37.17718011Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:39.204877742Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:39.205248807Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:40.705419058Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:40.705807524Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:18:40.859349873Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:41.309853841Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:41.310131001Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:41.487177518Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:41.517972532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:41.518303487Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:41.557163329Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:41.557397522Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:41.709116633Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:42.537327917Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:42.537768647Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:43.857972331Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:43.999612933Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:44.337955023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:45.118377125Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:45.118697373Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:45.358056145Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:45.358832408Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:45.726631791Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:45.727310298Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:45.926379043Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:45.927020049Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:46.380846993Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:46.611169877Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:46.612095689Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:46.986248667Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:47.088071718Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:47.088932254Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:48.254485854Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:48.558896828Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:48.644932416Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:48.645532754Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:49.200154796Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:49.554985806Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:49.555242876Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:49.651485137Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:18:50.859437453Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:51.700393547Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:51.701093137Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:51.90745717Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:51.908087732Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:51.982358644Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:52.118702565Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:52.119090067Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:52.577417825Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:52.69186274Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:52.999188956Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:52.999601397Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:53.315126312Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:53.897553822Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:54.980014021Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:54.980240725Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:55.342170349Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:55.342651468Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:55.435916519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:55.436108904Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:55.63945272Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:55.639902301Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:56.402141066Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:58.27242568Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:58.852874974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:59.296612655Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:59.296926798Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:59.714262844Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:59.7145369Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:59.870396386Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:18:59.870555172Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:19:00.859446615Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:00.930843324Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:00.977004581Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:00.977468287Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:01.823947067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:02.350587599Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:02.350766668Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:02.683857549Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:02.684843869Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:03.700642752Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:04.679558103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:04.720128074Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:07.179356962Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:07.210247654Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:07.234727581Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:07.234999984Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:07.271411737Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:07.27227403Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:08.022963145Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:08.128892005Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:09.71036927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:09.711612495Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:09.818550914Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:09.819335717Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:10.099480459Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:10.43639952Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:19:10.859425512Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:12.439034666Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:12.439572999Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:12.518115252Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:14.84331978Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:16.84434269Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:16.844799953Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:17.076338191Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:17.412511542Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:17.412917517Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:18.288294966Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:19.410157484Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:20.015953268Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:20.016479597Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:19:20.859439908Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:21.239943372Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:21.240290239Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:21.909030156Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:21.9092134Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:22.773567167Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:23.134749108Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:23.135012461Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:25.69860842Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:25.744227593Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:25.744479144Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:26.34045302Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:26.340705516Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:27.380650391Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:27.380926013Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:28.166508047Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:28.167376011Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:28.807018856Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:28.807415305Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:28.932227656Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:19:30.859435073Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:31.624481654Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:31.625165824Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:31.769131618Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:31.769375175Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:32.288137395Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:32.288551868Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:32.350054059Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:32.350435552Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:32.592791481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:32.593229253Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:32.905597046Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:33.214824643Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:33.215166599Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:33.795306226Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:33.796059246Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:33.815389483Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:33.816440225Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:33.818597113Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:34.132556118Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:34.132800453Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:35.312236491Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:35.577016604Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:35.57747302Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:36.688677781Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:36.725027168Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:36.924302547Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:37.165432715Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:37.165997416Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:39.127461623Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:39.127862658Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:40.689625144Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:40.69000686Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:19:40.879537919Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:41.350329349Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:41.350648666Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:41.486906688Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:41.528913483Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:41.529297495Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:41.573174228Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:41.573452062Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:41.752777634Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:42.212714003Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:42.213171307Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:43.86128605Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:43.999100983Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:44.338756154Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:44.549857113Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:44.550148268Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:45.358218825Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:45.358990937Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:45.727529875Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:45.728231327Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:45.919171135Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:45.91977354Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:46.376911417Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:46.588648524Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:46.589028178Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:46.984448945Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:47.069784119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:47.070139968Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:48.260089015Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:48.553813465Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:48.640517964Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:48.641165826Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:49.200680949Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:49.556708248Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:49.556957596Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:49.65064118Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:19:50.859437541Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:51.70940846Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:51.710124365Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:51.922726493Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:51.923201901Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:51.983881056Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:52.091096934Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:52.091610605Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:52.576597132Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:52.691365873Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:53.000882421Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:53.001331554Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:53.314274194Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:53.896269057Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:54.976835984Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:54.97706197Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:55.333761718Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:55.334390114Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:55.413180257Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:55.413335998Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:55.645339704Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:55.645800395Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:56.406293408Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:58.265951875Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:58.851567103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:59.281490127Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:59.281878923Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:59.716149686Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:59.71648837Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:59.889693774Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:19:59.889836371Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:20:00.859442331Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:00.867815025Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:00.868306207Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:00.930263184Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:03.589651242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:03.594079489Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:03.754567399Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:03.755500533Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:03.850166603Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:03.981612765Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:04.678775455Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:04.731305823Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:07.308619234Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:07.30997424Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:07.312955452Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:07.365301496Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:07.392286058Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:07.392574683Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:08.011694211Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:08.626700236Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:09.708985858Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:09.709580886Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:10.094280379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:10.094551609Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:10.106390083Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:10.45098088Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:20:10.8757586Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:12.478163159Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:12.478779514Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:12.51942327Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:14.84717718Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:16.854370682Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:16.854905503Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:17.077312693Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:17.453159484Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:17.453544935Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:18.165993454Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:19.410406227Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:19.689439928Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:19.689986994Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:20:20.859467511Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:21.291865276Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:21.292344049Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:21.910363884Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:21.910527252Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:22.767156524Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:23.135008254Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:23.135258038Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:25.697521519Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:25.744880579Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:25.745131016Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:26.272112208Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:26.272390273Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:27.381560567Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:27.3818177Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:28.177030114Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:28.177610528Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:28.749697231Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:28.750152786Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:28.936359258Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:20:30.859434835Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:31.743875773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:31.744562911Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:31.787832387Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:31.7882137Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:32.277467135Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:32.277876424Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:32.465148415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:32.465548256Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:32.593003594Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:32.593466567Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:32.905535365Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:33.223280966Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:33.223603455Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:33.795392566Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:33.796296247Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:33.816505081Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:33.817612928Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:33.819808776Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:34.132397245Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:34.132638438Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:35.313111373Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:35.577164295Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:35.577599956Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:36.664615245Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:36.665338772Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:36.953164194Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:37.161329258Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:37.161810532Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:39.272096256Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:39.272422665Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:40.741384772Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:40.741810866Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:20:40.859428724Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:41.508066979Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:41.610288082Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:41.610956015Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:41.739027987Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:41.741077261Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:41.745780797Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:41.746111239Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:41.75951767Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:42.552912884Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:42.55334223Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:43.95448071Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:44.000973114Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:44.341378498Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:45.137392762Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:45.137755513Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:45.358260831Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:45.359356356Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:45.764186188Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:45.764884895Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:45.924404029Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:45.925020268Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:46.467323321Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:46.615350802Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:46.615645216Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:46.982418741Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:47.066616125Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:47.066963096Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:48.605050267Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:48.701178153Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:48.91343086Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:49.214415417Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:49.690614914Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:49.741872034Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:49.743123795Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:20:50.860419429Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:51.450330513Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:51.7176171Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:51.718341146Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:51.905905766Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:51.906560634Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:51.983889914Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:52.075313361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:52.075744359Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:52.578389764Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:52.69431304Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:52.999284048Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:52.999803675Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:53.334972653Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:53.901363114Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:54.982777024Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:54.983007956Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:55.414978019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:55.415153275Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:55.471780046Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:55.47236803Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:55.672802743Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:55.673205284Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:56.402955198Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:58.264854709Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:58.857977906Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:59.117615738Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:59.117985856Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:59.71174952Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:59.712022419Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:59.88944001Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:20:59.889585817Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:21:00.859427326Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:00.931840894Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:01.277078967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:01.277567254Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:02.076930578Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:02.370190886Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:02.370380831Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:02.719752827Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:02.720429467Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:03.70700805Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:04.727823893Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:04.775074227Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:07.173414558Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:07.205587541Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:07.228690097Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:07.228962783Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:07.295960629Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:07.296706389Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:08.026887084Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:09.459841364Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:09.460224593Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:09.679178522Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:09.682257214Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:10.135984873Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:10.479190878Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:10.55092627Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:21:10.85977541Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:12.451518192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:12.452156537Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:12.597063625Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:14.844317036Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:16.724432124Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:16.724903007Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:17.076937896Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:17.403050654Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:17.403340736Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:18.442383216Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:19.411168322Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:19.684939886Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:19.685415193Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:21:20.859444431Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:21.22725649Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:21.227605862Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:21.903066096Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:21.903248575Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:22.773531102Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:23.138754762Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:23.139016565Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:25.697723596Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:25.738806197Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:25.73902643Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:26.271876838Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:26.272176207Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:27.38269773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:27.38296398Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:28.162621142Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:28.163180396Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:28.568983182Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:28.569503552Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:28.932518151Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:21:30.861212302Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:31.631054065Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:31.631593752Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:31.775120847Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:31.775351256Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:32.279154497Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:32.279532324Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:32.461575791Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:32.461987013Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:32.602141671Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:32.60258074Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:32.905620172Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:33.224509189Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:33.224840965Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:33.795337508Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:33.796133733Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:33.823188109Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:33.825085926Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:33.828196526Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:34.133546405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:34.133874297Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:35.313583433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:35.576799745Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:35.577251865Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:36.61432454Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:36.614760864Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:36.970453904Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:37.1609012Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:37.161366607Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:39.139113721Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:39.139579387Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:40.685391016Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:40.685696864Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:21:40.85942846Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:41.255096118Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:41.255523715Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:41.421878239Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:41.422244162Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:41.486596111Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:41.569605266Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:41.569965537Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:41.716964106Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:42.551965545Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:42.552413475Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:43.86172229Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:44.000053988Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:44.338825908Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:45.167724516Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:45.168021621Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:45.358260645Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:45.359204254Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:45.727283407Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:45.72797347Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:45.929248456Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:45.929895754Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:46.374338906Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:46.589888947Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:46.59022238Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:46.986346877Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:47.078708854Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:47.079109502Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:48.273187564Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:48.552920993Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:48.638176408Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:48.638617706Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:49.201371176Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:49.503792276Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:49.504036779Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:49.648157317Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:21:50.859426704Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:51.709285253Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:51.709994302Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:51.901311035Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:51.901862033Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:51.983930516Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:52.067796204Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:52.068213033Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:52.577469862Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:52.690742408Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:52.999083934Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:52.999527927Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:53.328835999Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:53.897201104Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:55.010580384Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:55.010803505Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:55.296072877Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:55.296514664Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:55.413480147Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:55.413623057Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:55.738356246Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:55.738791279Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:56.401854946Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:58.265054009Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:58.85175744Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:59.298260997Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:59.298556229Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:59.711420884Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:21:59.711738546Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:00.066320739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:00.066495224Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:22:00.859491215Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:00.930531006Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:01.022600466Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:01.023105769Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:01.595420442Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:02.387712007Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:02.387872074Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:02.672341715Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:02.673028048Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:03.69804128Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:04.691459685Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:04.720723905Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:07.173387131Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:07.205525542Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:07.230172031Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:07.230433816Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:07.266771319Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:07.267519048Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:08.012143196Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:08.381565217Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:08.935423786Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:08.936073966Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:09.740776936Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:09.74527323Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:10.02564415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:10.442356007Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:22:10.859422355Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:12.43620826Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:12.436783623Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:12.516860113Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:14.848840377Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:16.715220912Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:16.715714694Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:17.077114668Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:17.38653091Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:17.386854081Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:18.151437493Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:19.408744205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:19.690483913Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:19.691021218Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:22:20.859466944Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:21.914056705Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:21.914224934Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:21.945593348Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:21.946000035Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:22.766712274Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:23.135355525Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:23.135688934Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:25.697655397Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:25.727136924Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:25.727347268Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:26.286656256Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:26.286943453Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:27.383137571Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:27.383417083Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:28.160487978Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:28.161138485Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:28.576233953Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:28.576795539Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:28.938704735Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:22:30.859507705Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:31.630579172Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:31.631084683Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:31.769394719Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:31.769877853Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:32.294606931Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:32.295114568Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:32.466216012Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:32.46659932Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:32.591928643Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:32.592396116Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:32.905524328Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:33.213654464Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:33.214023511Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:33.795402793Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:33.796133273Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:33.816508711Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:33.817653763Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:33.819824761Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:34.133053958Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:34.133298791Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:35.323702164Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:35.576909707Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:35.57736095Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:36.6473771Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:36.64789592Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:36.920635494Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:37.165858838Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:37.166297678Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:39.204841525Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:39.205293276Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:40.695647653Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:40.695936009Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:22:40.859425821Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:41.260586396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:41.2608933Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:41.491657691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:41.511705939Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:41.512075753Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:41.565501242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:41.565819471Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:41.745131769Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:42.209501098Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:42.209989443Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:43.861641242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:43.998441023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:44.338566265Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:44.548278001Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:44.548549392Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:45.358161086Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:45.358943406Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:45.714171515Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:45.714851632Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:45.925104678Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:45.925694353Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:46.374999175Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:46.671724445Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:46.672179825Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:46.984723603Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:47.222506138Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:47.222838144Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:48.261782365Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:48.556478471Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:48.639305039Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:48.63994505Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:49.212538974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:49.504553324Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:49.504794081Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:49.638879537Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:22:50.859460784Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:51.723960672Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:51.72556121Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:51.920118296Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:51.921545175Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:51.982343875Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:52.125998088Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:52.127023662Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:52.580087052Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:52.715326623Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:52.999200848Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:52.999770175Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:53.33839971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:53.896352746Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:54.979254888Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:54.979546716Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:55.304950051Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:55.305360409Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:55.414943301Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:55.415092255Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:55.762754226Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:55.763181303Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:56.4034137Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:58.261282772Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:58.851641459Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:59.289193316Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:59.289496542Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:59.710231817Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:59.710486384Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:59.901656675Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:22:59.901869744Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:23:00.859439408Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:00.930277265Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:01.078321663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:01.078834976Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:01.945031301Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:02.369298815Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:02.36945958Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:02.672102117Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:02.67278458Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:03.697375908Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:04.687246181Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:04.720001422Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:07.177360283Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:07.207899231Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:07.228739697Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:07.229100736Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:07.269134281Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:07.269835701Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:08.011576122Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:08.206654381Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:08.881273195Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:08.887119418Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:09.744975178Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:09.745717011Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:10.013733513Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:10.444339666Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:23:10.859435352Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:12.442865193Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:12.4434102Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:12.521042601Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:14.849194533Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:16.603743461Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:16.604049356Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:16.839968929Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:16.840409781Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:17.076974568Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:18.183607773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:19.40886513Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:19.988089894Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:19.98858632Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:23:20.85944039Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:21.20831568Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:21.208683046Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:21.864947058Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:21.865118003Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:22.758256412Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:23.145487866Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:23.145771022Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:25.697384406Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:25.726575191Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:25.726854299Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:26.274703251Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:26.274977603Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:27.378875875Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:27.379193238Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:28.162283601Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:28.162952488Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:28.543487744Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:28.543897315Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:28.940595093Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:23:30.859416089Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:31.645114369Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:31.645680432Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:31.769650715Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:31.769947322Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:31.964159218Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:31.96451757Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:32.50114577Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:32.501517898Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:32.5923672Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:32.592802341Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:32.905929741Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:33.216017506Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:33.216350889Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:33.795552679Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:33.796301487Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:33.816648114Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:33.817847083Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:33.820082996Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:34.133170784Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:34.133404779Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:35.312752509Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:35.608910238Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:35.609368236Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:36.610579344Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:36.611162965Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:36.923785067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:37.169983856Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:37.170435383Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:39.097292931Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:39.09783273Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:40.71602046Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:40.716514781Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:23:40.86190415Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:41.270771696Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:41.27107916Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:41.49186869Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:41.51407536Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:41.514456075Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:41.564216787Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:41.56452679Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:41.709521795Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:42.556580906Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:42.557039868Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:43.855724475Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:43.999046097Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:44.339543704Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:45.132324034Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:45.132691482Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:45.358221664Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:45.359233925Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:45.723051197Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:45.723799846Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:45.930322783Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:45.930948683Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:46.376230133Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:46.616901236Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:46.617219652Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:46.985384349Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:47.223315998Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:47.223605585Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:48.26041414Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:48.512028166Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:48.512607584Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:48.554318237Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:49.200696622Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:49.556625405Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:49.556875771Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:49.646981109Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:23:50.859435273Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:51.629786192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:51.63047305Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:51.91193817Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:51.912462557Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:52.078690992Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:52.087845603Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:52.088330873Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:52.577331436Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:52.691465913Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:53.000760536Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:53.00124463Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:53.334307404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:53.896507384Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:54.971800576Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:54.971985729Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:55.33948889Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:55.340107475Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:55.43099744Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:55.4311585Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:55.8165857Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:55.817103246Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:56.40210573Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:58.305798277Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:58.8565174Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:59.285521445Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:59.285897038Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:59.722990645Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:59.723256488Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:59.88679399Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:23:59.886954854Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:24:00.859426924Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:00.930533538Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:01.005517331Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:01.005983637Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:01.495172761Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:02.433606601Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:02.433783668Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:02.674655475Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:02.675379054Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:03.701391098Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:04.676742376Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:04.72098558Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:07.172723095Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:07.202567375Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:07.220990371Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:07.221218647Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:07.267282889Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:07.268004953Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:08.011735222Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:08.533454529Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:09.446556604Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:09.447509623Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:09.72049532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:09.721322765Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:10.04998555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:10.431176207Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:24:10.859439384Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:12.430769509Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:12.431373788Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:12.595253144Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:14.842883782Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:16.847512591Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:16.848001987Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:17.038190105Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:17.038478284Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:17.073268224Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:18.243713952Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:19.408170134Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:19.6861192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:19.686579032Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:24:20.859504153Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:21.206646895Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:21.20699759Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:21.913272487Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:21.913457241Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:22.773304766Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:23.134937298Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:23.135199085Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:25.697584331Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:25.726548407Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:25.726805811Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:26.27199177Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:26.272300381Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:27.326688902Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:27.32698729Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:28.159987618Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:28.160562753Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:28.535979739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:28.536424041Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:28.93359239Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:24:30.859421212Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:31.769602813Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:31.769952085Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:31.962203029Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:31.962586816Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:32.473356147Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:32.473861297Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:32.493458911Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:32.493970707Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:32.594770903Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:32.59516096Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:32.905568291Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:33.213816088Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:33.214141226Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:33.795249125Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:33.796016794Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:33.820137889Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:33.821247935Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:33.823623409Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:34.150566386Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:34.150839197Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:35.31151275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:35.586174946Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:35.586632454Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:36.623826552Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:36.624316747Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:36.926702836Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:37.161199026Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:37.161616166Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:39.249795798Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:39.250195028Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:40.686457238Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:40.686736697Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:24:40.859428792Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:41.257326573Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:41.257617184Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:41.487524401Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:41.514136886Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:41.514570258Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:41.558727112Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:41.559004915Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:41.718220974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:42.552488642Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:42.553026826Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:43.874495597Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:44.000183613Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:44.271238925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:44.562599421Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:44.562934585Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:45.358221337Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:45.359025958Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:45.718864235Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:45.719580571Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:45.918518808Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:45.919138314Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:46.383077449Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:46.603827748Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:46.604224053Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:46.983503324Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:47.214348064Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:47.214767587Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:48.257172426Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:48.511118509Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:48.511686671Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:48.554348984Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:49.202714589Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:49.559292333Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:49.559552203Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:49.733436724Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:24:50.859427722Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:51.614462155Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:51.615201608Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:51.908233495Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:51.908911607Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:51.982306996Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:52.071234841Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:52.071632874Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:52.578142674Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:52.690799011Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:53.014888684Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:53.015714437Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:53.436092096Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:53.908041133Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:54.972491336Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:54.972703681Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:55.332928273Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:55.333343947Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:55.413520245Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:55.413680663Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:55.748299116Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:55.748780402Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:56.403469871Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:58.259376871Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:58.851552009Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:59.279658081Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:59.279967502Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:59.722533455Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:59.722815297Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:59.88659943Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:24:59.886769963Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:25:00.859425005Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:00.931739665Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:01.074925118Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:01.075395086Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:01.64994465Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:02.489427867Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:02.489623777Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:02.763849276Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:02.764767165Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:03.700861644Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:04.675216162Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:04.720961257Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:07.172842508Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:07.206393168Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:07.227397835Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:07.227729338Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:07.275910027Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:07.276754363Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:08.013254398Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:08.435072483Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:09.483102294Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:09.484417617Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:09.698021927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:09.698630058Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:10.156169445Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:10.461154818Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:25:10.859436007Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:12.466229993Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:12.466852303Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:12.596127757Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:14.841355017Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:16.720281628Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:16.720711263Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:17.077087291Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:17.441765083Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:17.442154119Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:18.136691819Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:19.408055645Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:19.684122974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:19.68464543Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:25:20.859431561Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:21.2078779Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:21.208209111Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:21.91852297Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:21.918812988Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:22.761168977Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:23.135412647Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:23.135708459Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:25.697555162Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:25.727247361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:25.727516115Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:26.271919588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:26.272219932Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:27.381408446Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:27.381728401Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:28.159924968Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:28.160365726Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:28.718372902Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:28.718958995Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:28.937919461Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:25:30.859502168Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:31.76970078Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:31.769972773Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:32.277491711Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:32.277908888Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:32.354005403Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:32.354400439Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:32.494814966Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:32.495351537Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:32.572894145Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:32.57333925Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:32.905634832Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:33.221278587Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:33.221606734Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:33.795492362Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:33.796238845Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:33.818164636Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:33.81964965Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:33.822500689Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:34.160070158Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:34.160332297Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:35.312630917Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:35.576600062Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:35.577103815Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:36.653502906Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:36.654027084Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:36.920538188Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:37.166653134Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:37.167158194Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:39.337043591Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:39.337392929Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:40.690099455Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:40.690406304Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:25:40.859475259Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:41.048751908Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:41.049006879Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:41.487809765Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:41.520569001Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:41.521274342Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:41.556037837Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:41.556389026Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:41.73984588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:42.557089495Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:42.557527395Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:43.856429927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:43.999704417Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:44.337531015Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:45.144033163Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:45.144351711Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:45.358281467Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:45.359048399Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:45.715877225Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:45.716614925Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:45.929429881Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:45.930091169Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:46.465221788Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:46.768320742Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:46.7688367Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:46.985793431Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:47.071788767Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:47.072133485Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:48.281094107Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:48.552398892Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:48.553831816Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:48.557692664Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:49.201356134Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:49.555354907Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:49.555582177Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:49.639898524Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:25:50.859438614Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:51.614633928Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:51.615346434Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:51.902230889Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:51.902873584Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:51.98862165Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:52.100394103Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:52.100854396Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:52.57876026Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:52.691900604Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:52.999440195Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:52.999900579Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:53.354851254Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:53.896401481Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:54.991255117Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:54.991459874Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:55.332832008Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:55.333279024Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:55.41287038Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:55.413028513Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:55.706033745Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:55.706498819Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:56.406444663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:58.254924386Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:58.851647626Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:59.313548566Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:59.313888761Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:59.732022394Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:59.732251376Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:59.925297557Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:25:59.925456072Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:26:00.859449896Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:00.931559773Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:01.027228625Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:01.027870907Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:02.032291927Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:02.345490118Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:02.345628012Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:02.673622569Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:02.674280803Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:03.700818663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:04.679289059Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:04.720981743Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:07.173407016Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:07.204214948Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:07.222860839Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:07.223082516Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:07.300565332Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:07.301318215Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:08.013926874Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:09.496467399Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:09.497060494Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:09.760841869Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:09.761354851Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:10.022338754Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:10.198171208Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:10.430396813Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:26:10.859449015Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:12.465358691Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:12.466023201Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:12.595255171Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:14.852780532Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:16.724315848Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:16.724775906Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:17.076480768Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:17.399232868Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:17.399513688Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:18.361793976Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:19.414558087Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:20.011833914Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:20.012300574Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:26:20.859448377Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:21.223406642Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:21.223785632Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:21.921141498Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:21.921326852Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:22.7736017Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:23.134306901Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:23.134545092Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:25.698173561Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:25.726343111Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:25.726600383Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:26.324651458Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:26.324929335Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:27.378422274Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:27.37874569Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:28.19616358Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:28.196823006Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:28.782403713Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:28.783197529Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:28.96144049Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:26:30.859504799Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:31.624856327Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:31.625497064Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:31.769593592Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:31.769849002Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:32.289990087Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:32.290500989Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:32.350792915Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:32.35117356Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:32.592475505Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:32.592946133Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:32.905880777Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:33.213565654Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:33.214012561Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:33.795238816Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:33.795971457Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:33.815399339Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:33.816491508Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:33.818619692Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:34.150338614Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:34.150579661Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:35.311076905Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:35.577266005Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:35.577742283Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:36.64407434Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:36.644542619Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:36.92560865Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:37.161041136Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:37.16146427Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:39.343243005Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:39.343915263Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:40.70129657Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:40.701556043Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:26:40.859421717Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:41.244336925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:41.244674826Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:41.490486544Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:41.525921151Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:41.526311403Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:41.56738183Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:41.567805145Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:41.719878322Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:42.568817742Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:42.569255217Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:43.864926957Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:43.997279026Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:44.27565651Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:45.126920253Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:45.127253476Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:45.358097334Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:45.358896838Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:45.71899183Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:45.719737598Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:45.920027795Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:45.920647445Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:46.373810241Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:46.771156723Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:46.771968234Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:46.981804565Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:47.069144907Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:47.06952927Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:48.258366017Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:48.555963161Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:48.644923961Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:48.645654056Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:49.200582415Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:49.554990543Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:49.555224583Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:49.650921958Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:26:50.859439106Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:51.629094027Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:51.629823605Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:51.940223021Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:51.940762249Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:51.989119901Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:52.109714575Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:52.11012966Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:52.578037205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:52.693695561Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:52.999353316Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:52.999822379Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:53.337866403Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:53.896798442Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:54.985605551Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:54.986118917Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:55.335298277Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:55.336185013Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:55.446076643Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:55.446393771Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:55.786550467Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:55.787793654Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:56.402044974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:58.289280638Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:58.851957642Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:59.088725754Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:59.089044828Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:59.729869739Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:59.730164203Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:59.870152359Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:26:59.870305955Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:27:00.859433497Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:00.930298717Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:01.007235338Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:01.007757855Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:01.816980427Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:02.345391156Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:02.345549258Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:02.6996019Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:02.700440862Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:03.698246849Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:04.679312555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:04.720304812Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:07.186373984Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:07.217243304Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:07.244023896Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:07.244353946Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:07.272862382Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:07.273698333Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:08.020474238Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:08.096810129Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:09.526065174Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:09.52705728Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:09.715029554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:09.715627901Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:10.033398464Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:10.430432711Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:27:10.859431859Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:12.463194612Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:12.463828963Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:12.520704634Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:14.8479412Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:16.727946424Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:16.728332649Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:17.07573496Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:17.447208335Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:17.447532849Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:18.287123814Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:19.404871379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:19.680614012Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:19.681122293Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:27:20.859453428Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:21.209155332Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:21.209531629Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:21.931256219Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:21.931404014Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:22.772412671Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:23.155189604Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:23.155484187Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:25.699797529Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:25.726426971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:25.726835636Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:26.271979117Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:26.272259979Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:27.311370917Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:27.311694976Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:28.161816668Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:28.162301234Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:28.731919097Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:28.732480589Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:28.977985885Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:27:30.85940808Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:31.619220144Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:31.61988608Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:31.769880874Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:31.770148579Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:32.274557556Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:32.274937418Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:32.354005096Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:32.354355917Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:32.592885626Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:32.593282835Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:32.905575714Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:33.22534074Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:33.225703876Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:33.795266769Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:33.795997561Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:33.816178439Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:33.817328914Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:33.81949801Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:34.168442445Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:34.168759757Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:35.324228643Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:35.577068216Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:35.577512344Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:36.650243936Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:36.650765787Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:36.920606002Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:37.160745502Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:37.161195972Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:39.102532369Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:39.102922576Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:40.744475205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:40.744913909Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:27:40.859441004Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:41.29588669Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:41.29623251Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:41.4874339Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:41.521074921Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:41.521448299Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:41.561658588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:41.561991349Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:41.721836242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:42.568407296Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:42.568982784Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:43.858999998Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:43.997813473Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:44.340890117Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:44.54932426Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:44.549955486Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:45.358088771Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:45.358888474Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:45.712631897Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:45.713311592Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:45.928758775Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:45.929353871Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:46.380631122Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:46.596281709Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:46.596649811Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:46.982607464Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:47.058714876Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:47.059018307Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:48.256150716Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:48.561927775Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:48.64941421Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:48.649915645Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:49.202573796Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:49.493016724Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:49.493212981Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:49.644919114Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:27:50.859424443Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:51.625974632Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:51.627038706Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:51.891280092Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:51.891830691Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:51.945173887Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:52.103691941Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:52.104198536Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:52.576911956Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:52.690991432Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:52.99921079Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:52.999654845Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:53.331056242Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:53.896312951Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:54.988870079Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:54.989089375Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:55.332536833Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:55.332987058Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:55.434065538Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:55.434221847Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:55.645534501Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:55.645978773Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:56.404173083Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:58.287987495Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:58.851527554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:59.094707709Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:59.095024163Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:59.72322967Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:59.723562888Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:59.870196586Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:27:59.870358404Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:28:00.859426626Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:00.930278099Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:00.981571361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:00.982028413Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:01.71888982Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:02.36979685Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:02.36995987Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:02.674646662Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:02.675284896Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:03.69911277Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:04.673334586Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:04.720844686Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:07.192717591Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:07.229600929Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:07.252246685Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:07.25246611Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:07.293763336Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:07.294498406Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:08.011715484Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:08.292358708Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:09.422239911Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:09.422625297Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:09.692273074Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:09.692745195Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:10.013511899Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:10.445176442Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:28:10.872742395Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:12.438238419Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:12.438898691Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:12.596775863Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:14.845492362Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:16.838632364Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:16.839091043Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:17.076966814Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:17.411382882Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:17.41173296Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:18.244361012Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:19.403324695Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:19.700836226Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:19.70130953Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:28:20.859430167Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:21.207958199Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:21.208344201Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:21.910675111Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:21.910842153Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:22.760431646Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:23.134802958Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:23.135087092Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:25.698813097Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:25.726802678Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:25.727014355Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:26.272030857Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:26.272285507Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:27.379553986Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:27.3799139Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:28.174128042Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:28.175083479Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:28.612545572Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:28.613016515Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:28.934190143Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:28:30.859441866Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:31.647256575Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:31.647802027Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:31.769753722Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:31.76999221Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:32.284353538Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:32.284822201Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:32.476744776Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:32.477106298Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:32.592392549Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:32.592858025Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:32.905543474Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:33.214770743Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:33.21509113Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:33.795326867Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:33.796093517Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:33.816253901Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:33.817373792Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:33.819533964Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:34.150890379Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:34.151121206Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:35.312393504Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:35.578393535Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:35.578858798Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:36.612137809Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:36.61251381Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:36.927907434Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:37.165858636Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:37.166334458Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:39.377103802Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:39.378388769Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:40.686009346Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:40.686284063Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:28:40.859434811Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:41.186816404Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:41.187113968Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:41.489203407Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:41.506635942Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:41.506994156Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:41.55938369Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:41.559770001Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:41.756597767Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:42.539023509Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:42.539460234Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:43.856391488Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:43.998596816Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:44.337718847Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:45.128442446Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:45.12879923Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:45.358072323Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:45.358850775Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:45.73661913Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:45.737374901Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:45.923880116Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:45.924441288Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:46.377345365Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:46.592101511Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:46.592476985Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:46.982417139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:47.060389509Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:47.060712562Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:48.275244787Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:48.553784411Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:48.649859339Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:48.650451116Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:49.20259119Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:49.503506234Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:49.50373885Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:49.637597342Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:28:50.859509364Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:51.623881981Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:51.624650049Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:51.911898023Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:51.912549833Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:52.082797036Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:52.083197835Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:52.083535175Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:52.578631197Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:52.691168337Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:53.000783171Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:53.001248957Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:53.32024101Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:53.896479778Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:54.975126798Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:54.975320147Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:55.304794109Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:55.305360944Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:55.413186414Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:55.413315472Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:55.762328952Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:55.762868067Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:56.416020301Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:58.30672598Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:58.870540829Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:59.299102765Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:59.299421113Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:59.733349212Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:59.733621541Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:59.87005223Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:28:59.87021964Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:29:00.859500433Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:00.931741553Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:00.993330306Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:00.993803918Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:01.758104682Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:02.368339387Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:02.368507462Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:02.677702871Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:02.678566559Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:03.700953028Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:04.68032999Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:04.720058762Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:07.171569432Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:07.201303356Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:07.219366884Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:07.21960067Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:07.266574681Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:07.267310166Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:08.011880931Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:08.161517288Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:09.413872803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:09.414261972Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:09.714908305Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:09.717496961Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:10.020929317Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:10.431699912Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:29:10.859435223Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:12.442296912Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:12.442918459Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:12.519904179Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:14.846099075Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:16.839687411Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:16.840114246Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:17.074464868Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:17.44115494Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:17.441461211Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:18.170089864Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:19.407437311Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:19.994348249Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:19.994913506Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:29:20.859505149Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:21.247184136Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:21.247472649Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:22.030261361Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:22.030420399Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:22.774071971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:23.13610396Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:23.13635231Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:25.697610021Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:25.726652499Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:25.726872083Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:26.340095181Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:26.340384738Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:27.378647197Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:27.378973366Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:28.169422555Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:28.170263673Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:28.63583143Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:28.636353455Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:28.947641252Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:29:30.859434749Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:31.625926908Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:31.62642961Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:31.769092148Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:31.769345402Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:31.961892152Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:31.962263795Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:32.46112193Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:32.461586977Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:32.593121375Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:32.593514489Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:32.905716274Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:33.213570653Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:33.213908826Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:33.795251741Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:33.795970826Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:33.815170863Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:33.816298365Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:33.818430293Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:34.151428827Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:34.152082453Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:35.309953524Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:35.577051573Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:35.577537437Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:36.641656007Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:36.642179595Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:36.919915977Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:37.16168335Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:37.16212821Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:39.430403013Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:39.431280345Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:40.685781401Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:40.686041066Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:29:40.859480446Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:41.253554235Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:41.253877085Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:41.488284777Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:41.531617081Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:41.532020131Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:41.556455822Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:41.556790851Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:41.716617036Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:42.211655009Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:42.21211116Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:43.856052475Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:43.998278971Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:44.339303461Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:44.549544659Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:44.549901339Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:45.358231076Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:45.359265006Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:45.721288762Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:45.72198059Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:45.931116282Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:45.931887518Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:46.374489088Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:46.601318657Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:46.601638682Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:46.982629005Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:47.084522199Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:47.084866486Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:48.256433328Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:48.554772826Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:48.657900592Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:48.658363189Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:49.201992652Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:49.557951527Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:49.558215378Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:49.642049061Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:29:50.859431631Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:51.614264026Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:51.6149451Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:51.916712037Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:51.917208268Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:51.989679669Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:52.067460461Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:52.067867827Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:52.57846585Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:52.691451942Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:53.00223436Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:53.002718305Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:53.306048572Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:53.897248517Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:54.973548748Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:54.973784885Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:55.296066393Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:55.29655451Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:55.412811467Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:55.4129491Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:55.776114814Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:55.776485765Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:56.402768586Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:58.297743823Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:58.851542517Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:59.284465574Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:59.284866472Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:59.723476643Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:59.723794664Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:59.869881077Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:29:59.870040677Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:30:00.859432754Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:00.930486009Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:01.016438455Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:01.016920076Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:03.554469602Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:03.587937647Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:03.794320251Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:03.812706446Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:04.178686981Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:04.827213466Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:04.876614111Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:04.975581027Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:07.226401915Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:07.275311504Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:07.30290371Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:07.303200287Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:07.310394737Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:07.31151926Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:08.022225832Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:08.918258003Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:08.918641064Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:09.732647214Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:09.733345222Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:10.037345693Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:10.462076495Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:30:10.859433797Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:11.036594797Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:12.439965806Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:12.440591649Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:12.517654388Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:14.844707581Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:16.716098925Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:16.716538768Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:17.076183282Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:17.404688492Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:17.405082026Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:18.282616562Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:19.407893982Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:19.689778447Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:19.690294929Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:30:20.859427338Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:21.288707986Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:21.28918726Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:21.867358366Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:21.86752445Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:22.764558107Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:23.135237057Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:23.135523446Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:25.708035536Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:25.728461786Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:25.728917195Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:26.295503504Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:26.29596373Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:27.37986649Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:27.380165388Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:28.163036174Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:28.163624078Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:28.655548914Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:28.656254548Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:28.96281539Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:30:30.85945589Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:31.649365072Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:31.64998474Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:31.769151311Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:31.769412732Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:32.111514493Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:32.111988883Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:32.455714652Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:32.456074073Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:32.595313541Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:32.596142567Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:32.905520242Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:33.21543604Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:33.215845291Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:33.795238478Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:33.796058937Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:33.815694229Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:33.816867341Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:33.819023188Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:34.150505546Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:34.15079153Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:35.3121792Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:35.577547827Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:35.578146283Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:36.973549012Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:37.191470029Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:37.19206227Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:37.769592189Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:37.770163681Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:39.787420743Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:39.787978763Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:40.688049935Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:40.688333071Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:30:40.85942799Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:41.252640424Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:41.252968998Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:41.48904171Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:41.530566698Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:41.531186992Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:41.561711698Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:41.561973285Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:41.725536995Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:42.564528907Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:42.565004701Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:43.877264002Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:44.002285523Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:44.338804496Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:45.133908999Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:45.134218636Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:45.358075964Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:45.358817165Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:45.719520188Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:45.720403345Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:45.927405104Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:45.927991537Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:46.379591839Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:46.705891993Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:46.706369016Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:46.986899082Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:47.074720098Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:47.075030447Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:48.258682545Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:48.554975014Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:48.739239816Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:48.740259298Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:49.217623573Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:49.560050286Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:49.560319668Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:49.672542363Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:30:50.859424037Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:51.666100276Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:51.666980804Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:51.905771097Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:51.906317455Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:51.985246403Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:52.07379703Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:52.074213647Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:52.587932366Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:52.705203676Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:52.999841031Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:53.000305657Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:53.336341225Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:53.901469399Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:55.184086849Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:55.184332104Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:55.304171928Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:55.304590704Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:55.412797764Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:55.412947036Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:55.776741251Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:55.777174337Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:56.40509984Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:58.38674223Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:58.996503741Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:59.19221974Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:59.193048435Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:59.744929699Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:59.745445867Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:59.896619301Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:30:59.914904858Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:31:00.859417479Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:00.931423766Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:01.410748084Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:01.411230016Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:02.264101825Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:02.368519428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:02.368735931Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:02.675719248Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:02.676414839Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:03.698178545Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:04.684337638Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:04.722103751Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:07.175484926Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:07.206821964Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:07.226243964Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:07.226479203Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:07.270214644Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:07.270914078Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:08.01604155Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:08.818136422Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:08.818481262Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:09.713098247Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:09.71658517Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:10.027135067Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:10.426631478Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:10.434510464Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:31:10.859462094Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:12.448498213Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:12.449120628Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:12.520984981Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:14.854112408Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:16.715158982Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:16.715548863Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:17.077598996Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:17.419729623Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:17.420176147Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:18.15907384Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:19.408891071Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:19.691612157Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:19.692138336Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:31:20.860771054Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:21.223057097Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:21.223376193Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:21.906163543Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:21.906342691Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:22.773259744Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:23.134509515Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:23.13501Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:25.697052596Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:25.726677021Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:25.726960935Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:26.271455627Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:26.271771336Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:27.38249147Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:27.382823158Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:28.161759858Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:28.162369591Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:28.711174141Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:28.711793514Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:28.933853112Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:31:30.859424706Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:31.64940293Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:31.649961302Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:31.769250371Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:31.769509838Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:31.9556448Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:31.956063872Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:32.459501782Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:32.459900356Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:32.600044638Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:32.600505821Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:32.905528872Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:33.214742627Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:33.215071431Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:33.795381098Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:33.796113809Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:33.815585328Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:33.816696767Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:33.818881712Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:34.150939397Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:34.151230737Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:35.310245879Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:35.577005469Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:35.577452196Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:36.657565907Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:36.658078229Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:36.92414357Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:37.160794473Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:37.161173546Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:39.178696863Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:39.179512434Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:40.700379738Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:40.700650684Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:31:40.859442509Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:41.262952001Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:41.26321822Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:41.490615929Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:41.509343105Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:41.50975582Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:41.565274254Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:41.56555674Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:41.720502312Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:42.546910942Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:42.54737471Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:43.860793879Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:44.003360806Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:44.337561433Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:45.148311591Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:45.148564033Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:45.358110699Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:45.358912235Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:45.714239618Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:45.714942341Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:45.949821582Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:45.950432707Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:46.374505733Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:46.598265588Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:46.59864247Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:46.985440174Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:47.099642554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:47.099978377Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:48.257907663Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:48.555576885Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:48.671560704Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:48.672185431Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:49.20459774Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:49.557782933Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:49.558060533Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:49.638800447Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:31:50.859440724Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:51.628389826Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:51.629210892Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:51.915629647Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:51.916314659Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:51.986421357Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:52.073241933Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:52.073633296Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:52.578190733Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:52.69682905Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:52.999091466Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:52.999545215Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:53.300510969Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:53.896310487Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:54.973749579Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:54.974015308Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:55.332364932Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:55.332850212Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:55.412786621Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:55.412972123Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:55.770330528Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:55.770784621Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:56.405396558Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:58.292393422Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:58.865888575Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:59.313565716Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:59.314074308Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:59.723127996Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:31:59.723429358Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:00.050428344Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:00.050593046Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:32:00.85947805Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:00.930756885Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:01.035304897Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:01.035771046Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:01.846652173Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:02.346677653Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:02.346831809Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:02.742961137Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:02.743853778Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:03.697778401Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:04.680366517Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:04.72664962Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:07.19273892Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:07.22605711Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:07.247408247Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:07.247690573Z caller=manager.go:402 component="rule manager" group=aggregate_container_resources msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:07.29322989Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:07.294030508Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.180:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:08.011766554Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:10443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:08.506689704Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.78.254:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:09.399685205Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:09.399976472Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.206:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:09.693117921Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:09.693580805Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.71.128:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:10.018169811Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.76.216:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:10.433523812Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.133:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:32:10.859621596Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:12.447762569Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:12.448364948Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.64.11:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:12.516544277Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.66:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:14.842142755Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.123:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:16.839648934Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:16.840132227Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.216:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:17.076053184Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.251:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:17.417418795Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:17.41771778Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.127:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:18.455978033Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.73.251:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:19.408436732Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.38:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:19.994048313Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:19.994571696Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.80:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:32:20.859434358Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:21.22486105Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:21.225192727Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.216:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:21.905581631Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:21.905791599Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.75.47:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:22.760089809Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.74.37:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:23.134933048Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:23.135204804Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.76.203:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:25.701852825Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.71:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:25.738632142Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:25.738842286Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.72:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:26.34296721Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:26.343256701Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.80:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:28.165050515Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:28.165631198Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.231:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:28.73011305Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:28.730630159Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.53:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:28.939060841Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.127:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:32:30.85944458Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:31.61984867Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:31.620396426Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.85:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:31.773512851Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:31.773793116Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.2:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:32.281423143Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:32.281845406Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.66.71:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:32.349334107Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:32.349736918Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.38:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:32.606003803Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:32.606475336Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.72.79:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:32.905514475Z caller=manager.go:402 component="rule manager" group="Node rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:33.213830913Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:33.21414291Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.77.169:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:33.795257116Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:33.796016074Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:33.81616374Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:33.817475995Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:33.819652113Z caller=manager.go:402 component="rule manager" group=kubernetes-rules msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:34.150813583Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:34.151068092Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.64.11:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:35.312474546Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.79.85:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:35.579755191Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:35.580211417Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.70.104:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:36.596728428Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:36.597076409Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.73.251:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:36.924420139Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.74:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:37.160927089Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:37.161350838Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.74.37:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:37.302745714Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:37.303026496Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.65.66:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:39.276261533Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:39.276823767Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.78.254:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:40.68605017Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:40.68627169Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.84:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:32:40.859452046Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:41.263356507Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:41.263847044Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.60:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:41.429135275Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:41.429496902Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.71.128:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:41.487421589Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.231:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:41.562757264Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:41.563048382Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.65.74:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:41.724528411Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.69.53:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:42.209790828Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:42.210208554Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.66:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:43.860598174Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.78.254:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:44.001712995Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.77.169:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:44.358005738Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.73.206:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:44.548563417Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:44.548879321Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.74.37:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:45.358140534Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:45.358894152Z caller=manager.go:402 component="rule manager" group="Metrics rules" msg="rule sample appending failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:45.713113017Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:45.713816959Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.76.203:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:45.973016678Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:45.973657193Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.77.169:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:46.375329296Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.68.60:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:46.60382958Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:46.604180163Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.231:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:46.985962493Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.66.80:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:47.078209192Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:47.078535495Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.79:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:48.258466099Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:48.554200932Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.70.104:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:48.59211966Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:48.592531226Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.127:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:49.200456018Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.71.128:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:49.554873504Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:49.555099005Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.66.71:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:49.640979794Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.53:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:32:50.859509398Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:51.652160996Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:51.65290667Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.68.133:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:51.914360338Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:51.915055362Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.79.2:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:51.986444111Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.65.74:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:52.090559576Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:52.091015321Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.69.84:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:52.578276238Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=http://10.130.2.190:8080/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:52.691354936Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.128.2.113:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:52.999002463Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:52.999449399Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.206:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:53.329631356Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.251:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:53.897534054Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.72.79:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:54.979848558Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:54.980056709Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.78.254:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:55.332282121Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:55.332750262Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.60:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:55.43253698Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:55.432713454Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.180:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:55.771961861Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:55.772403868Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.75.47:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:56.404280894Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-controllers target=https://172.31.73.251:8444/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:58.308254039Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.78.254:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:58.851489768Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.129.0.14:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:59.283224229Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:59.283552633Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.70.104:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:59.724734141Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:59.725009455Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.68.133:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:59.870350604Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:32:59.870511962Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.79.85:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=error ts=2018-09-21T07:33:00.859749526Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:33:00.948317601Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.2.131:9443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:33:01.178553834Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:33:01.181261119Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.65.74:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:33:02.418991908Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:33:02.419293457Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-nodes target=https://172.31.73.38:10250/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:33:02.472409689Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-apiservers target=https://172.31.65.74:443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:33:02.724610169Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:33:02.726783435Z caller=scrape.go:717 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://172.31.72.72:10250/metrics/cadvisor msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:33:03.713440107Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.69.84:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:33:04.677529225Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-nodes-exporter target=http://172.31.75.47:9100/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed" >level=warn ts=2018-09-21T07:33:04.721714599Z caller=scrape.go:713 component="scrape manager" scrape_pool=kubernetes-service-endpoints target=https://10.130.2.192:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed"
<b>You cannot view the attachment while viewing its details because your browser does not support IFRAMEs. <a href="attachment.cgi?id=1485428">View the attachment on a separate page</a>.</b>
View Attachment As Raw
Actions:
View
Attachments on
bug 1631667
: 1485428