Attachment 1487375 Details for Bug 1624493 – output of: master-logs etcd etcd &> etcd.log

output of: master-logs etcd etcd &> etcd.log

etcd.log (text/plain), 48.02 KB, created by Randolph Morgan on 2018-09-26 16:54:05 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Randolph Morgan

Created: 2018-09-26 16:54:05 UTC

Size: 48.02 KB

patch

obsolete

>2018-09-25 19:04:34.706069 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=https://192.168.105.111:2379
>2018-09-25 19:04:34.706189 I | pkg/flags: recognized and used environment variable ETCD_CERT_FILE=/etc/etcd/server.crt
>2018-09-25 19:04:34.706196 I | pkg/flags: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=true
>2018-09-25 19:04:34.706206 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd/
>2018-09-25 19:04:34.706211 I | pkg/flags: recognized and used environment variable ETCD_DEBUG=False
>2018-09-25 19:04:34.706218 I | pkg/flags: recognized and used environment variable ETCD_ELECTION_TIMEOUT=2500
>2018-09-25 19:04:34.706232 I | pkg/flags: recognized and used environment variable ETCD_HEARTBEAT_INTERVAL=500
>2018-09-25 19:04:34.706243 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.105.111:2380
>2018-09-25 19:04:34.706252 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
>2018-09-25 19:04:34.706257 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-1
>2018-09-25 19:04:34.706262 I | pkg/flags: recognized and used environment variable ETCD_KEY_FILE=/etc/etcd/server.key
>2018-09-25 19:04:34.706271 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=https://192.168.105.111:2379
>2018-09-25 19:04:34.706277 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=https://192.168.105.111:2380
>2018-09-25 19:04:34.706289 I | pkg/flags: recognized and used environment variable ETCD_NAME=openshift-master1.chem.byu.edu
>2018-09-25 19:04:34.706296 I | pkg/flags: recognized and used environment variable ETCD_PEER_CERT_FILE=/etc/etcd/peer.crt
>2018-09-25 19:04:34.706302 I | pkg/flags: recognized and used environment variable ETCD_PEER_CLIENT_CERT_AUTH=true
>2018-09-25 19:04:34.706308 I | pkg/flags: recognized and used environment variable ETCD_PEER_KEY_FILE=/etc/etcd/peer.key
>2018-09-25 19:04:34.706313 I | pkg/flags: recognized and used environment variable ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/ca.crt
>2018-09-25 19:04:34.706322 I | pkg/flags: recognized and used environment variable ETCD_QUOTA_BACKEND_BYTES=4294967296
>2018-09-25 19:04:34.706335 I | pkg/flags: recognized and used environment variable ETCD_TRUSTED_CA_FILE=/etc/etcd/ca.crt
>2018-09-25 19:04:34.706355 W | pkg/flags: unrecognized environment variable ETCD_INITIAL_CLUSTER=
>2018-09-25 19:04:34.706378 I | etcdmain: etcd Version: 3.2.22
>2018-09-25 19:04:34.706383 I | etcdmain: Git SHA: 1674e682f
>2018-09-25 19:04:34.706387 I | etcdmain: Go Version: go1.8.7
>2018-09-25 19:04:34.706391 I | etcdmain: Go OS/Arch: linux/amd64
>2018-09-25 19:04:34.706395 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4
>2018-09-25 19:04:34.706479 W | etcdmain: found invalid file/dir openshift-backup-post-3.0-20180517150431 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706488 W | etcdmain: found invalid file/dir openshift-backup-post-3.0-20180517164019 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706492 W | etcdmain: found invalid file/dir openshift-backup-post-3.0-20180518084412 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706495 W | etcdmain: found invalid file/dir openshift-backup-post-3.0-20180518094405 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706499 W | etcdmain: found invalid file/dir openshift-backup-post-3.0-20180521085141 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706503 W | etcdmain: found invalid file/dir openshift-backup-post-3.0-20180524123509 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706507 W | etcdmain: found invalid file/dir openshift-backup-post-3.0-20180524131649 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706510 W | etcdmain: found invalid file/dir openshift-backup-post-3.0-20180524142319 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706515 W | etcdmain: found invalid file/dir openshift-backup-post-3.0-20180524145613 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706518 W | etcdmain: found invalid file/dir openshift-backup-post-3.0-20180524151229 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706522 W | etcdmain: found invalid file/dir openshift-backup-post-3.0-20180524164156 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706527 W | etcdmain: found invalid file/dir openshift-backup-post-3.0-20180524170256 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706530 W | etcdmain: found invalid file/dir openshift-backup-post-3.0-20180823085435 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706537 W | etcdmain: found invalid file/dir openshift-backup-pre-migration20180517144930 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706541 W | etcdmain: found invalid file/dir openshift-backup-pre-upgrade-20180517150325 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706545 W | etcdmain: found invalid file/dir openshift-backup-pre-upgrade-20180517163914 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706549 W | etcdmain: found invalid file/dir openshift-backup-pre-upgrade-20180518084307 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706553 W | etcdmain: found invalid file/dir openshift-backup-pre-upgrade-20180518094300 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706557 W | etcdmain: found invalid file/dir openshift-backup-pre-upgrade-20180521085033 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706561 W | etcdmain: found invalid file/dir openshift-backup-pre-upgrade-20180524123347 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706564 W | etcdmain: found invalid file/dir openshift-backup-pre-upgrade-20180524131534 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706568 W | etcdmain: found invalid file/dir openshift-backup-pre-upgrade-20180524142209 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706574 W | etcdmain: found invalid file/dir openshift-backup-pre-upgrade-20180524145504 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706577 W | etcdmain: found invalid file/dir openshift-backup-pre-upgrade-20180524151119 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706581 W | etcdmain: found invalid file/dir openshift-backup-pre-upgrade-20180524164047 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706585 W | etcdmain: found invalid file/dir openshift-backup-pre-upgrade-20180524170147 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706588 W | etcdmain: found invalid file/dir openshift-backup-pre-upgrade-20180823085137 under data dir /var/lib/etcd/ (Ignore this if you are upgrading etcd)
>2018-09-25 19:04:34.706595 N | etcdmain: the server is already initialized as member before, starting as etcd member...
>2018-09-25 19:04:34.706617 I | embed: peerTLS: cert = /etc/etcd/peer.crt, key = /etc/etcd/peer.key, ca = , trusted-ca = /etc/etcd/ca.crt, client-cert-auth = true
>2018-09-25 19:04:34.707314 I | embed: listening for peers on https://192.168.105.111:2380
>2018-09-25 19:04:34.707376 I | embed: listening for client requests on 192.168.105.111:2379
>2018-09-25 19:04:35.135375 I | etcdserver: recovered store from snapshot at index 218736194
>2018-09-25 19:04:35.308957 I | mvcc: restore compact to 51441010
>2018-09-25 19:04:35.592568 I | etcdserver: name = openshift-master1.chem.byu.edu
>2018-09-25 19:04:35.592604 I | etcdserver: data dir = /var/lib/etcd/
>2018-09-25 19:04:35.592611 I | etcdserver: member dir = /var/lib/etcd/member
>2018-09-25 19:04:35.592616 I | etcdserver: heartbeat = 500ms
>2018-09-25 19:04:35.592620 I | etcdserver: election = 2500ms
>2018-09-25 19:04:35.592631 I | etcdserver: snapshot count = 100000
>2018-09-25 19:04:35.592646 I | etcdserver: advertise client URLs = https://192.168.105.111:2379
>2018-09-25 19:04:36.014914 I | etcdserver: restarting member b86402c9b4aca9cc in cluster e2968a57f86740c7 at commit index 218812345
>2018-09-25 19:04:36.019757 I | raft: b86402c9b4aca9cc became follower at term 516
>2018-09-25 19:04:36.019795 I | raft: newRaft b86402c9b4aca9cc [peers: [cc2e639c5cf7ad2,b86402c9b4aca9cc,f853b05503ea4e1f], term: 516, commit: 218812345, applied: 218736194, lastindex: 218812345, lastterm: 516]
>2018-09-25 19:04:36.024380 I | etcdserver/api: enabled capabilities for version 3.2
>2018-09-25 19:04:36.024405 I | etcdserver/membership: added member b86402c9b4aca9cc [https://192.168.105.111:2380] to cluster e2968a57f86740c7 from store
>2018-09-25 19:04:36.024418 I | etcdserver/membership: added member cc2e639c5cf7ad2 [https://192.168.105.112:2380] to cluster e2968a57f86740c7 from store
>2018-09-25 19:04:36.025485 I | etcdserver/membership: added member f853b05503ea4e1f [https://192.168.105.113:2380] to cluster e2968a57f86740c7 from store
>2018-09-25 19:04:36.025501 I | etcdserver/membership: set the cluster version to 3.2 from store
>2018-09-25 19:04:36.179691 I | mvcc: restore compact to 51441010
>2018-09-25 19:04:36.204575 W | auth: simple token is not cryptographically signed
>2018-09-25 19:04:36.230196 I | rafthttp: starting peer cc2e639c5cf7ad2...
>2018-09-25 19:04:36.230249 I | rafthttp: started HTTP pipelining with peer cc2e639c5cf7ad2
>2018-09-25 19:04:36.236313 I | rafthttp: started peer cc2e639c5cf7ad2
>2018-09-25 19:04:36.236341 I | rafthttp: added peer cc2e639c5cf7ad2
>2018-09-25 19:04:36.236351 I | rafthttp: starting peer f853b05503ea4e1f...
>2018-09-25 19:04:36.236384 I | rafthttp: started HTTP pipelining with peer f853b05503ea4e1f
>2018-09-25 19:04:36.236873 I | rafthttp: started peer f853b05503ea4e1f
>2018-09-25 19:04:36.236882 I | rafthttp: added peer f853b05503ea4e1f
>2018-09-25 19:04:36.236903 I | etcdserver: starting server... [version: 3.2.22, cluster version: 3.2]
>2018-09-25 19:04:36.238991 I | rafthttp: started streaming with peer f853b05503ea4e1f (writer)
>2018-09-25 19:04:36.239025 I | rafthttp: started streaming with peer cc2e639c5cf7ad2 (writer)
>2018-09-25 19:04:36.239038 I | rafthttp: started streaming with peer cc2e639c5cf7ad2 (writer)
>2018-09-25 19:04:36.239111 I | rafthttp: started streaming with peer f853b05503ea4e1f (writer)
>2018-09-25 19:04:36.239125 I | embed: ClientTLS: cert = /etc/etcd/server.crt, key = /etc/etcd/server.key, ca = , trusted-ca = /etc/etcd/ca.crt, client-cert-auth = true
>2018-09-25 19:04:36.239160 I | rafthttp: started streaming with peer f853b05503ea4e1f (stream MsgApp v2 reader)
>2018-09-25 19:04:36.240939 I | rafthttp: started streaming with peer cc2e639c5cf7ad2 (stream MsgApp v2 reader)
>2018-09-25 19:04:36.241670 I | rafthttp: started streaming with peer cc2e639c5cf7ad2 (stream Message reader)
>2018-09-25 19:04:36.242279 I | rafthttp: started streaming with peer f853b05503ea4e1f (stream Message reader)
>2018-09-25 19:04:39.099609 I | rafthttp: peer cc2e639c5cf7ad2 became active
>2018-09-25 19:04:39.099630 I | rafthttp: established a TCP streaming connection with peer cc2e639c5cf7ad2 (stream MsgApp v2 reader)
>2018-09-25 19:04:39.100496 I | rafthttp: established a TCP streaming connection with peer cc2e639c5cf7ad2 (stream Message reader)
>2018-09-25 19:04:39.131077 I | rafthttp: established a TCP streaming connection with peer cc2e639c5cf7ad2 (stream Message writer)
>2018-09-25 19:04:39.131247 I | rafthttp: established a TCP streaming connection with peer cc2e639c5cf7ad2 (stream MsgApp v2 writer)
>2018-09-25 19:04:40.025902 I | raft: b86402c9b4aca9cc is starting a new election at term 516
>2018-09-25 19:04:40.025973 I | raft: b86402c9b4aca9cc became candidate at term 517
>2018-09-25 19:04:40.025987 I | raft: b86402c9b4aca9cc received MsgVoteResp from b86402c9b4aca9cc at term 517
>2018-09-25 19:04:40.025997 I | raft: b86402c9b4aca9cc [logterm: 516, index: 218812345] sent MsgVote request to cc2e639c5cf7ad2 at term 517
>2018-09-25 19:04:40.026017 I | raft: b86402c9b4aca9cc [logterm: 516, index: 218812345] sent MsgVote request to f853b05503ea4e1f at term 517
>2018-09-25 19:04:40.046398 I | raft: b86402c9b4aca9cc received MsgVoteResp rejection from cc2e639c5cf7ad2 at term 517
>2018-09-25 19:04:40.046437 I | raft: b86402c9b4aca9cc [quorum:2] has received 1 MsgVoteResp votes and 1 vote rejections
>2018-09-25 19:04:40.255566 I | rafthttp: peer f853b05503ea4e1f became active
>2018-09-25 19:04:40.255707 I | rafthttp: established a TCP streaming connection with peer f853b05503ea4e1f (stream MsgApp v2 reader)
>2018-09-25 19:04:40.255976 I | rafthttp: established a TCP streaming connection with peer f853b05503ea4e1f (stream Message reader)
>2018-09-25 19:04:40.265067 I | etcdserver: b86402c9b4aca9cc initialzed peer connection; fast-forwarding 3 ticks (election ticks 5) with 2 active peer(s)
>2018-09-25 19:04:40.268084 I | raft: b86402c9b4aca9cc received MsgVoteResp from f853b05503ea4e1f at term 517
>2018-09-25 19:04:40.268115 I | raft: b86402c9b4aca9cc [quorum:2] has received 2 MsgVoteResp votes and 1 vote rejections
>2018-09-25 19:04:40.268128 I | raft: b86402c9b4aca9cc became leader at term 517
>2018-09-25 19:04:40.268142 I | raft: raft.node: b86402c9b4aca9cc elected leader b86402c9b4aca9cc at term 517
>2018-09-25 19:04:40.276540 I | etcdserver: published {Name:openshift-master1.chem.byu.edu ClientURLs:[https://192.168.105.111:2379]} to cluster e2968a57f86740c7
>2018-09-25 19:04:40.276615 I | embed: ready to serve client requests
>2018-09-25 19:04:40.276969 I | embed: serving client requests on 192.168.105.111:2379
>2018-09-25 19:04:40.358793 I | rafthttp: established a TCP streaming connection with peer f853b05503ea4e1f (stream Message writer)
>2018-09-25 19:04:40.359648 I | rafthttp: established a TCP streaming connection with peer f853b05503ea4e1f (stream MsgApp v2 writer)
>WARNING: 2018/09/25 19:04:40 Failed to dial 192.168.105.111:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.
>2018-09-25 19:04:41.801627 W | etcdserver: apply entries took too long [138.149457ms for 1 entries]
>2018-09-25 19:04:41.801647 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 19:04:50.639245 W | etcdserver: apply entries took too long [103.253873ms for 1 entries]
>2018-09-25 19:04:50.639265 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 19:05:09.033613 W | etcdserver: apply entries took too long [118.568008ms for 1 entries]
>2018-09-25 19:05:09.033634 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 19:08:14.347856 I | mvcc: store.index: compact 51441886
>2018-09-25 19:08:14.352628 I | mvcc: finished scheduled compaction at 51441886 (took 2.61135ms)
>2018-09-25 19:08:17.046275 W | etcdserver: apply entries took too long [112.695589ms for 1 entries]
>2018-09-25 19:08:17.046298 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 19:08:25.373970 W | etcdserver: apply entries took too long [112.355205ms for 1 entries]
>2018-09-25 19:08:25.374003 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 19:13:14.400906 I | mvcc: store.index: compact 51442625
>2018-09-25 19:13:14.405127 I | mvcc: finished scheduled compaction at 51442625 (took 2.395732ms)
>2018-09-25 19:14:51.909054 W | etcdserver: apply entries took too long [112.046039ms for 1 entries]
>2018-09-25 19:14:51.909076 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 19:15:30.944708 W | etcdserver: apply entries took too long [125.182965ms for 1 entries]
>2018-09-25 19:15:30.944730 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 19:18:14.432615 I | mvcc: store.index: compact 51443379
>2018-09-25 19:18:14.438713 I | mvcc: finished scheduled compaction at 51443379 (took 2.659988ms)
>2018-09-25 19:22:14.890712 W | etcdserver: apply entries took too long [121.766747ms for 1 entries]
>2018-09-25 19:22:14.890734 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 19:22:59.801759 W | etcdserver: apply entries took too long [159.952802ms for 1 entries]
>2018-09-25 19:22:59.801782 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 19:23:14.473966 I | mvcc: store.index: compact 51444025
>2018-09-25 19:23:14.477997 I | mvcc: finished scheduled compaction at 51444025 (took 2.544824ms)
>2018-09-25 19:28:14.500505 I | mvcc: store.index: compact 51444586
>2018-09-25 19:28:14.504958 I | mvcc: finished scheduled compaction at 51444586 (took 2.360837ms)
>2018-09-25 19:32:37.070259 W | etcdserver: apply entries took too long [134.658194ms for 1 entries]
>2018-09-25 19:32:37.070309 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 19:33:14.539925 I | mvcc: store.index: compact 51445138
>2018-09-25 19:33:14.543793 I | mvcc: finished scheduled compaction at 51445138 (took 2.264229ms)
>2018-09-25 19:38:14.553717 I | mvcc: store.index: compact 51445738
>2018-09-25 19:38:14.558118 I | mvcc: finished scheduled compaction at 51445738 (took 2.816692ms)
>2018-09-25 19:41:05.682552 W | etcdserver: apply entries took too long [112.043513ms for 1 entries]
>2018-09-25 19:41:05.682581 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 19:41:29.359131 W | etcdserver: apply entries took too long [100.944958ms for 1 entries]
>2018-09-25 19:41:29.359171 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 19:42:19.133162 W | wal: sync duration of 1.152746689s, expected less than 1s
>2018-09-25 19:42:19.133235 W | etcdserver: failed to send out heartbeat on time (exceeded the 500ms timeout for 155.8216ms)
>2018-09-25 19:42:19.133245 W | etcdserver: server is likely overloaded
>2018-09-25 19:42:19.133251 W | etcdserver: failed to send out heartbeat on time (exceeded the 500ms timeout for 155.840105ms)
>2018-09-25 19:42:19.133255 W | etcdserver: server is likely overloaded
>2018-09-25 19:42:22.735304 I | raft: b86402c9b4aca9cc [logterm: 517, index: 218820887, vote: b86402c9b4aca9cc] ignored MsgVote from f853b05503ea4e1f [logterm: 517, index: 218820867] at term 517: lease is not expired (remaining ticks: 4)
>2018-09-25 19:42:23.903089 I | raft: b86402c9b4aca9cc [term: 517] received a MsgApp message with higher term from f853b05503ea4e1f [term: 518]
>2018-09-25 19:42:23.903148 I | raft: b86402c9b4aca9cc became follower at term 518
>2018-09-25 19:42:23.903164 I | raft: found conflict at index 218820868 [existing term: 517, conflicting term: 518]
>2018-09-25 19:42:23.903169 I | raft: truncate the unstable entries before index 218820868
>2018-09-25 19:42:23.903192 I | raft: raft.node: b86402c9b4aca9cc changed leader from b86402c9b4aca9cc to f853b05503ea4e1f at term 518
>2018-09-25 19:42:24.238985 W | wal: sync duration of 5.10562618s, expected less than 1s
>2018-09-25 19:42:24.777436 W | etcdserver: apply entries took too long [538.07417ms for 3 entries]
>2018-09-25 19:42:24.777457 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 19:42:24.777944 W | etcdserver: failed to revoke 29cc66121e6dfc31 ("lease not found")
>2018-09-25 19:42:24.778128 W | etcdserver: failed to revoke 29cc66121e6dfc31 ("lease not found")
>2018-09-25 19:42:25.001325 W | etcdserver: failed to revoke 29cc66121e6dfc31 ("lease not found")
>2018-09-25 19:42:29.500791 W | etcdserver: failed to revoke 29cc66121e6dfc31 ("etcdserver: request timed out")
>2018-09-25 19:42:30.001325 W | etcdserver: failed to revoke 29cc66121e6dfc31 ("etcdserver: request timed out")
>2018-09-25 19:42:30.001352 W | etcdserver: failed to revoke 29cc66121e6dfc3b ("etcdserver: request timed out")
>2018-09-25 19:42:30.502008 W | etcdserver: failed to revoke 29cc66121e6dfc3b ("etcdserver: request timed out")
>2018-09-25 19:42:30.502081 W | etcdserver: failed to revoke 29cc66121e6dfc31 ("etcdserver: request timed out")
>2018-09-25 19:42:31.002420 W | etcdserver: failed to revoke 29cc66121e6dfc31 ("etcdserver: request timed out")
>2018-09-25 19:42:31.002464 W | etcdserver: failed to revoke 29cc66121e6dfc3b ("etcdserver: request timed out")
>2018-09-25 19:42:31.502746 W | etcdserver: failed to revoke 29cc66121e6dfc3b ("etcdserver: request timed out")
>2018-09-25 19:42:31.502884 W | etcdserver: failed to revoke 29cc66121e6dfc31 ("etcdserver: request timed out")
>2018-09-25 19:42:32.003126 W | etcdserver: failed to revoke 29cc66121e6dfc3b ("etcdserver: request timed out")
>2018-09-25 19:42:32.003191 W | etcdserver: failed to revoke 29cc66121e6dfc31 ("etcdserver: request timed out")
>2018-09-25 19:42:32.503543 W | etcdserver: failed to revoke 29cc66121e6dfc31 ("etcdserver: request timed out")
>2018-09-25 19:42:32.503567 W | etcdserver: failed to revoke 29cc66121e6dfc3b ("etcdserver: request timed out")
>2018-09-25 19:42:33.004021 W | etcdserver: failed to revoke 29cc66121e6dfc31 ("etcdserver: request timed out")
>2018-09-25 19:42:33.004060 W | etcdserver: failed to revoke 29cc66121e6dfc3b ("etcdserver: request timed out")
>2018-09-25 19:42:33.025783 I | raft: b86402c9b4aca9cc is starting a new election at term 518
>2018-09-25 19:42:33.025827 I | raft: b86402c9b4aca9cc became candidate at term 519
>2018-09-25 19:42:33.025849 I | raft: b86402c9b4aca9cc received MsgVoteResp from b86402c9b4aca9cc at term 519
>2018-09-25 19:42:33.025862 I | raft: b86402c9b4aca9cc [logterm: 518, index: 218820883] sent MsgVote request to f853b05503ea4e1f at term 519
>2018-09-25 19:42:33.025904 I | raft: b86402c9b4aca9cc [logterm: 518, index: 218820883] sent MsgVote request to cc2e639c5cf7ad2 at term 519
>2018-09-25 19:42:33.025913 I | raft: raft.node: b86402c9b4aca9cc lost leader f853b05503ea4e1f at term 519
>2018-09-25 19:42:33.505148 W | etcdserver: failed to revoke 29cc66121e6dfc31 ("etcdserver: request timed out")
>2018-09-25 19:42:33.505234 W | etcdserver: failed to revoke 29cc66121e6dfc3b ("etcdserver: request timed out")
>2018-09-25 19:42:35.095671 I | raft: b86402c9b4aca9cc [logterm: 518, index: 218820883, vote: b86402c9b4aca9cc] rejected MsgVote from cc2e639c5cf7ad2 [logterm: 518, index: 218820883] at term 519
>2018-09-25 19:42:37.525828 I | raft: b86402c9b4aca9cc is starting a new election at term 519
>2018-09-25 19:42:37.525948 I | raft: b86402c9b4aca9cc became candidate at term 520
>2018-09-25 19:42:37.525981 I | raft: b86402c9b4aca9cc received MsgVoteResp from b86402c9b4aca9cc at term 520
>2018-09-25 19:42:37.526042 I | raft: b86402c9b4aca9cc [logterm: 518, index: 218820883] sent MsgVote request to f853b05503ea4e1f at term 520
>2018-09-25 19:42:37.526093 I | raft: b86402c9b4aca9cc [logterm: 518, index: 218820883] sent MsgVote request to cc2e639c5cf7ad2 at term 520
>2018-09-25 19:42:37.790331 I | raft: b86402c9b4aca9cc received MsgVoteResp from cc2e639c5cf7ad2 at term 520
>2018-09-25 19:42:37.790448 I | raft: b86402c9b4aca9cc [quorum:2] has received 2 MsgVoteResp votes and 0 vote rejections
>2018-09-25 19:42:37.790489 I | raft: b86402c9b4aca9cc became leader at term 520
>2018-09-25 19:42:37.790542 I | raft: raft.node: b86402c9b4aca9cc elected leader b86402c9b4aca9cc at term 520
>2018-09-25 19:42:37.790986 I | raft: b86402c9b4aca9cc [logterm: 520, index: 218820888, vote: b86402c9b4aca9cc] rejected MsgVote from f853b05503ea4e1f [logterm: 518, index: 218820889] at term 520
>2018-09-25 19:42:40.723549 W | etcdserver: timed out waiting for read index response
>2018-09-25 19:43:14.641785 I | mvcc: store.index: compact 51446300
>2018-09-25 19:43:14.645699 I | mvcc: finished scheduled compaction at 51446300 (took 2.356339ms)
>2018-09-25 19:44:00.259413 W | etcdserver: apply entries took too long [150.424951ms for 1 entries]
>2018-09-25 19:44:00.259431 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 19:46:03.207814 W | etcdserver: apply entries took too long [113.318093ms for 1 entries]
>2018-09-25 19:46:03.207852 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 19:46:36.923498 W | etcdserver: apply entries took too long [116.635158ms for 1 entries]
>2018-09-25 19:46:36.923526 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 19:47:36.086873 W | etcdserver: apply entries took too long [101.272088ms for 1 entries]
>2018-09-25 19:47:36.086894 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 19:48:14.696277 I | mvcc: store.index: compact 51446865
>2018-09-25 19:48:14.699999 I | mvcc: finished scheduled compaction at 51446865 (took 2.248153ms)
>2018-09-25 19:49:59.317735 W | etcdserver: apply entries took too long [136.858451ms for 1 entries]
>2018-09-25 19:49:59.317760 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 19:52:09.759364 W | etcdserver: apply entries took too long [150.679483ms for 1 entries]
>2018-09-25 19:52:09.759390 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 19:53:14.717938 I | mvcc: store.index: compact 51447471
>2018-09-25 19:53:14.721923 I | mvcc: finished scheduled compaction at 51447471 (took 2.26268ms)
>2018-09-25 19:58:14.735645 I | mvcc: store.index: compact 51448062
>2018-09-25 19:58:14.739701 I | mvcc: finished scheduled compaction at 51448062 (took 2.633307ms)
>2018-09-25 19:59:53.754345 W | etcdserver: apply entries took too long [105.141656ms for 1 entries]
>2018-09-25 19:59:53.755263 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 20:00:40.333709 W | etcdserver: apply entries took too long [140.775124ms for 1 entries]
>2018-09-25 20:00:40.333738 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 20:03:02.231319 W | etcdserver: apply entries took too long [105.883754ms for 1 entries]
>2018-09-25 20:03:02.231353 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 20:03:14.746447 I | mvcc: store.index: compact 51448606
>2018-09-25 20:03:14.750086 I | mvcc: finished scheduled compaction at 51448606 (took 2.113302ms)
>2018-09-25 20:06:48.309566 W | etcdserver: apply entries took too long [129.421574ms for 1 entries]
>2018-09-25 20:06:48.309595 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 20:08:14.759341 I | mvcc: store.index: compact 51449168
>2018-09-25 20:08:14.763853 I | mvcc: finished scheduled compaction at 51449168 (took 2.586032ms)
>2018-09-25 20:10:36.039760 I | raft: b86402c9b4aca9cc [logterm: 520, index: 218826822, vote: b86402c9b4aca9cc] ignored MsgVote from cc2e639c5cf7ad2 [logterm: 520, index: 218826818] at term 520: lease is not expired (remaining ticks: 4)
>2018-09-25 20:10:36.041736 I | raft: b86402c9b4aca9cc [term: 520] received a MsgApp message with higher term from cc2e639c5cf7ad2 [term: 521]
>2018-09-25 20:10:36.041759 I | raft: b86402c9b4aca9cc became follower at term 521
>2018-09-25 20:10:36.041768 I | raft: found conflict at index 218826819 [existing term: 520, conflicting term: 521]
>2018-09-25 20:10:36.041774 I | raft: truncate the unstable entries before index 218826819
>2018-09-25 20:10:36.041782 I | raft: raft.node: b86402c9b4aca9cc changed leader from b86402c9b4aca9cc to cc2e639c5cf7ad2 at term 521
>2018-09-25 20:10:36.235429 W | wal: sync duration of 3.166484008s, expected less than 1s
>2018-09-25 20:10:38.543480 W | wal: sync duration of 1.228485558s, expected less than 1s
>2018-09-25 20:10:43.087463 W | etcdserver: timed out waiting for read index response
>2018-09-25 20:10:45.309312 W | etcdserver: failed to revoke 4e1f66121e7dbbad ("etcdserver: request timed out")
>2018-09-25 20:10:45.809776 W | etcdserver: failed to revoke 4e1f66121e7dbbad ("etcdserver: request timed out")
>2018-09-25 20:10:51.025858 I | raft: b86402c9b4aca9cc is starting a new election at term 521
>2018-09-25 20:10:51.025914 I | raft: b86402c9b4aca9cc became candidate at term 522
>2018-09-25 20:10:51.025925 I | raft: b86402c9b4aca9cc received MsgVoteResp from b86402c9b4aca9cc at term 522
>2018-09-25 20:10:51.025935 I | raft: b86402c9b4aca9cc [logterm: 521, index: 218826867] sent MsgVote request to f853b05503ea4e1f at term 522
>2018-09-25 20:10:51.025943 I | raft: b86402c9b4aca9cc [logterm: 521, index: 218826867] sent MsgVote request to cc2e639c5cf7ad2 at term 522
>2018-09-25 20:10:51.025951 I | raft: raft.node: b86402c9b4aca9cc lost leader cc2e639c5cf7ad2 at term 522
>2018-09-25 20:10:51.158518 I | raft: b86402c9b4aca9cc received MsgVoteResp from f853b05503ea4e1f at term 522
>2018-09-25 20:10:51.158541 I | raft: b86402c9b4aca9cc [quorum:2] has received 2 MsgVoteResp votes and 0 vote rejections
>2018-09-25 20:10:51.158559 I | raft: b86402c9b4aca9cc became leader at term 522
>2018-09-25 20:10:51.158594 I | raft: raft.node: b86402c9b4aca9cc elected leader b86402c9b4aca9cc at term 522
>2018-09-25 20:10:57.909396 W | etcdserver: timed out waiting for read index response
>2018-09-25 20:13:14.774165 I | mvcc: store.index: compact 51449775
>2018-09-25 20:13:14.778735 I | mvcc: finished scheduled compaction at 51449775 (took 2.556502ms)
>2018-09-25 20:16:54.785232 W | etcdserver: apply entries took too long [122.06882ms for 1 entries]
>2018-09-25 20:16:54.785265 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 20:18:14.791230 I | mvcc: store.index: compact 51450321
>2018-09-25 20:18:14.794916 I | mvcc: finished scheduled compaction at 51450321 (took 2.161364ms)
>2018-09-25 20:18:57.504050 W | etcdserver: apply entries took too long [134.058793ms for 1 entries]
>2018-09-25 20:18:57.504091 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 20:19:40.520751 W | etcdserver: apply entries took too long [119.564517ms for 1 entries]
>2018-09-25 20:19:40.520778 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 20:23:14.813250 I | mvcc: store.index: compact 51450883
>2018-09-25 20:23:14.817400 I | mvcc: finished scheduled compaction at 51450883 (took 2.268587ms)
>2018-09-25 20:25:58.714456 W | etcdserver: apply entries took too long [137.604738ms for 1 entries]
>2018-09-25 20:25:58.714480 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 20:27:47.549764 W | etcdserver: apply entries took too long [123.158685ms for 1 entries]
>2018-09-25 20:27:47.549786 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 20:28:14.829473 I | mvcc: store.index: compact 51451467
>2018-09-25 20:28:14.833515 I | mvcc: finished scheduled compaction at 51451467 (took 2.190093ms)
>2018-09-25 20:28:24.296026 W | etcdserver: apply entries took too long [121.727345ms for 1 entries]
>2018-09-25 20:28:24.296050 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 20:28:43.018688 W | etcdserver: apply entries took too long [101.83747ms for 1 entries]
>2018-09-25 20:28:43.018712 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 20:29:23.055721 W | etcdserver: apply entries took too long [138.327615ms for 1 entries]
>2018-09-25 20:29:23.055746 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 20:33:14.870123 I | mvcc: store.index: compact 51452034
>2018-09-25 20:33:14.873997 I | mvcc: finished scheduled compaction at 51452034 (took 2.383826ms)
>2018-09-25 20:38:14.883064 I | mvcc: store.index: compact 51452585
>2018-09-25 20:38:14.886995 I | mvcc: finished scheduled compaction at 51452585 (took 1.994592ms)
>2018-09-25 20:39:33.811332 W | etcdserver: apply entries took too long [153.194404ms for 1 entries]
>2018-09-25 20:39:33.811373 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 20:41:39.491315 W | etcdserver: apply entries took too long [116.931264ms for 1 entries]
>2018-09-25 20:41:39.491335 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 20:43:14.898083 I | mvcc: store.index: compact 51453159
>2018-09-25 20:43:14.902566 I | mvcc: finished scheduled compaction at 51453159 (took 2.548374ms)
>2018-09-25 20:47:55.039327 W | etcdserver: apply entries took too long [111.777197ms for 1 entries]
>2018-09-25 20:47:55.039384 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 20:48:14.999922 I | mvcc: store.index: compact 51453747
>2018-09-25 20:48:15.004449 I | mvcc: finished scheduled compaction at 51453747 (took 2.455765ms)
>2018-09-25 20:49:34.352304 W | etcdserver: apply entries took too long [115.02449ms for 1 entries]
>2018-09-25 20:49:34.352324 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 20:51:16.067610 W | etcdserver: apply entries took too long [116.264096ms for 1 entries]
>2018-09-25 20:51:16.067691 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 20:52:41.035910 W | etcdserver: apply entries took too long [110.817789ms for 1 entries]
>2018-09-25 20:52:41.035952 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 20:53:15.135067 I | mvcc: store.index: compact 51454321
>2018-09-25 20:53:15.139133 I | mvcc: finished scheduled compaction at 51454321 (took 2.095304ms)
>2018-09-25 20:55:32.081304 I | etcdserver: start to snapshot (applied: 218836195, lastsnap: 218736194)
>2018-09-25 20:55:32.975787 I | etcdserver: saved snapshot at index 218836195
>2018-09-25 20:55:32.976170 I | etcdserver: compacted raft log at 218831195
>2018-09-25 20:55:36.317894 I | pkg/fileutil: purged file /var/lib/etcd/member/snap/00000000000001c2-000000000d038bbe.snap successfully
>2018-09-25 20:57:20.023056 W | etcdserver: apply entries took too long [111.633056ms for 1 entries]
>2018-09-25 20:57:20.023095 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 20:58:15.166568 I | mvcc: store.index: compact 51454881
>2018-09-25 20:58:15.172570 I | mvcc: finished scheduled compaction at 51454881 (took 3.374612ms)
>2018-09-25 21:02:59.700293 W | etcdserver: apply entries took too long [101.620805ms for 1 entries]
>2018-09-25 21:02:59.700315 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 21:03:15.190757 I | mvcc: store.index: compact 51455456
>2018-09-25 21:03:15.195079 I | mvcc: finished scheduled compaction at 51455456 (took 2.401277ms)
>2018-09-25 21:05:23.762846 W | etcdserver: apply entries took too long [102.637483ms for 1 entries]
>2018-09-25 21:05:23.762950 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 21:07:58.142996 W | etcdserver: apply entries took too long [104.154644ms for 1 entries]
>2018-09-25 21:07:58.143032 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 21:08:15.208876 I | mvcc: store.index: compact 51456054
>2018-09-25 21:08:15.212654 I | mvcc: finished scheduled compaction at 51456054 (took 2.328222ms)
>2018-09-25 21:10:29.059505 W | etcdserver: apply entries took too long [103.612815ms for 1 entries]
>2018-09-25 21:10:29.059528 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 21:13:15.249057 I | mvcc: store.index: compact 51456599
>2018-09-25 21:13:15.253142 I | mvcc: finished scheduled compaction at 51456599 (took 2.302986ms)
>2018-09-25 21:14:11.507578 W | wal: sync duration of 2.229668406s, expected less than 1s
>2018-09-25 21:14:11.507699 W | etcdserver: failed to send out heartbeat on time (exceeded the 500ms timeout for 1.231028641s)
>2018-09-25 21:14:11.507707 W | etcdserver: server is likely overloaded
>2018-09-25 21:14:11.507712 W | etcdserver: failed to send out heartbeat on time (exceeded the 500ms timeout for 1.231044401s)
>2018-09-25 21:14:11.507716 W | etcdserver: server is likely overloaded
>2018-09-25 21:14:13.540393 I | raft: b86402c9b4aca9cc [logterm: 522, index: 218840127, vote: b86402c9b4aca9cc] ignored MsgVote from cc2e639c5cf7ad2 [logterm: 522, index: 218840121] at term 522: lease is not expired (remaining ticks: 3)
>2018-09-25 21:14:15.231872 W | wal: sync duration of 3.724075585s, expected less than 1s
>2018-09-25 21:14:15.231949 W | etcdserver: failed to send out heartbeat on time (exceeded the 500ms timeout for 2.724224119s)
>2018-09-25 21:14:15.231956 W | etcdserver: server is likely overloaded
>2018-09-25 21:14:15.231962 W | etcdserver: failed to send out heartbeat on time (exceeded the 500ms timeout for 2.724237668s)
>2018-09-25 21:14:15.231966 W | etcdserver: server is likely overloaded
>2018-09-25 21:14:15.232784 I | raft: b86402c9b4aca9cc [term: 522] received a MsgAppResp message with higher term from cc2e639c5cf7ad2 [term: 523]
>2018-09-25 21:14:15.232858 I | raft: b86402c9b4aca9cc became follower at term 523
>2018-09-25 21:14:15.232871 I | raft: raft.node: b86402c9b4aca9cc changed leader from b86402c9b4aca9cc to cc2e639c5cf7ad2 at term 523
>2018-09-25 21:14:15.233415 I | raft: b86402c9b4aca9cc [term: 523] ignored a MsgAppResp message with lower term from f853b05503ea4e1f [term: 522]
>2018-09-25 21:14:15.256992 I | raft: b86402c9b4aca9cc [term: 523] ignored a MsgAppResp message with lower term from f853b05503ea4e1f [term: 522]
>2018-09-25 21:14:15.257044 I | raft: b86402c9b4aca9cc [term: 523] ignored a MsgHeartbeatResp message with lower term from f853b05503ea4e1f [term: 522]
>2018-09-25 21:14:15.320600 I | raft: b86402c9b4aca9cc [term: 523] ignored a MsgHeartbeatResp message with lower term from f853b05503ea4e1f [term: 522]
>2018-09-25 21:14:15.420498 W | etcdserver: apply entries took too long [188.320011ms for 1 entries]
>2018-09-25 21:14:15.420719 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 21:14:17.540037 I | raft: b86402c9b4aca9cc [term: 523] received a MsgVote message with higher term from cc2e639c5cf7ad2 [term: 524]
>2018-09-25 21:14:17.540092 I | raft: b86402c9b4aca9cc became follower at term 524
>2018-09-25 21:14:17.540106 I | raft: b86402c9b4aca9cc [logterm: 522, index: 218840133, vote: 0] rejected MsgVote from cc2e639c5cf7ad2 [logterm: 522, index: 218840121] at term 524
>2018-09-25 21:14:17.540117 I | raft: raft.node: b86402c9b4aca9cc lost leader cc2e639c5cf7ad2 at term 524
>2018-09-25 21:14:19.574589 I | raft: b86402c9b4aca9cc [term: 524] ignored a MsgVote message with lower term from f853b05503ea4e1f [term: 523]
>2018-09-25 21:14:20.525897 I | raft: b86402c9b4aca9cc is starting a new election at term 524
>2018-09-25 21:14:20.525942 I | raft: b86402c9b4aca9cc became candidate at term 525
>2018-09-25 21:14:20.525951 I | raft: b86402c9b4aca9cc received MsgVoteResp from b86402c9b4aca9cc at term 525
>2018-09-25 21:14:20.525964 I | raft: b86402c9b4aca9cc [logterm: 522, index: 218840133] sent MsgVote request to cc2e639c5cf7ad2 at term 525
>2018-09-25 21:14:20.525972 I | raft: b86402c9b4aca9cc [logterm: 522, index: 218840133] sent MsgVote request to f853b05503ea4e1f at term 525
>2018-09-25 21:14:20.530493 I | raft: b86402c9b4aca9cc received MsgVoteResp from f853b05503ea4e1f at term 525
>2018-09-25 21:14:20.530514 I | raft: b86402c9b4aca9cc [quorum:2] has received 2 MsgVoteResp votes and 0 vote rejections
>2018-09-25 21:14:20.530561 I | raft: b86402c9b4aca9cc became leader at term 525
>2018-09-25 21:14:20.530574 I | raft: raft.node: b86402c9b4aca9cc elected leader b86402c9b4aca9cc at term 525
>2018-09-25 21:14:20.588824 W | etcdserver: failed to revoke 29cc66121e6edfaf ("lease not found")
>2018-09-25 21:14:25.232333 W | etcdserver: timed out waiting for read index response
>2018-09-25 21:18:15.276918 I | mvcc: store.index: compact 51457168
>2018-09-25 21:18:15.281042 I | mvcc: finished scheduled compaction at 51457168 (took 2.201154ms)
>2018-09-25 21:21:33.627332 W | etcdserver: apply entries took too long [126.843725ms for 1 entries]
>2018-09-25 21:21:33.627403 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 21:23:15.335322 I | mvcc: store.index: compact 51457749
>2018-09-25 21:23:15.340160 I | mvcc: finished scheduled compaction at 51457749 (took 2.664915ms)
>2018-09-25 21:28:15.358156 I | mvcc: store.index: compact 51458343
>2018-09-25 21:28:15.362525 I | mvcc: finished scheduled compaction at 51458343 (took 2.477202ms)
>2018-09-25 21:29:57.189264 W | etcdserver: apply entries took too long [138.180677ms for 1 entries]
>2018-09-25 21:29:57.189288 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 21:32:23.360468 W | etcdserver: apply entries took too long [121.094051ms for 1 entries]
>2018-09-25 21:32:23.360486 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 21:32:43.377843 W | etcdserver: apply entries took too long [108.087037ms for 1 entries]
>2018-09-25 21:32:43.377864 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 21:33:15.386333 I | mvcc: store.index: compact 51458914
>2018-09-25 21:33:15.393324 I | mvcc: finished scheduled compaction at 51458914 (took 4.163955ms)
>2018-09-25 21:38:15.454889 I | mvcc: store.index: compact 51459490
>2018-09-25 21:38:15.459022 I | mvcc: finished scheduled compaction at 51459490 (took 2.230698ms)
>2018-09-25 21:39:40.048589 W | etcdserver: apply entries took too long [150.244305ms for 1 entries]
>2018-09-25 21:39:40.048608 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 21:39:54.215887 W | etcdserver: apply entries took too long [129.090763ms for 1 entries]
>2018-09-25 21:39:54.215910 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 21:43:15.470512 I | mvcc: store.index: compact 51460067
>2018-09-25 21:43:15.475574 I | mvcc: finished scheduled compaction at 51460067 (took 2.939729ms)
>2018-09-25 21:44:09.990430 W | etcdserver: apply entries took too long [113.48635ms for 1 entries]
>2018-09-25 21:44:09.990457 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 21:48:15.516048 I | mvcc: store.index: compact 51460642
>2018-09-25 21:48:15.520050 I | mvcc: finished scheduled compaction at 51460642 (took 2.196945ms)
>2018-09-25 21:49:31.699402 I | wal: segmented wal file /var/lib/etcd/member/wal/0000000000000a3c-000000000d0b5967.wal is created
>2018-09-25 21:49:36.319842 I | pkg/fileutil: purged file /var/lib/etcd/member/wal/0000000000000a37-000000000d088657.wal successfully
>2018-09-25 21:49:59.715944 W | etcdserver: apply entries took too long [144.141681ms for 1 entries]
>2018-09-25 21:49:59.716009 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 21:50:08.443030 W | etcdserver: apply entries took too long [111.133206ms for 1 entries]
>2018-09-25 21:50:08.443052 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 21:51:22.546216 W | etcdserver: apply entries took too long [129.769394ms for 1 entries]
>2018-09-25 21:51:22.546235 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 21:53:02.926289 W | etcdserver: apply entries took too long [107.342206ms for 1 entries]
>2018-09-25 21:53:02.926387 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 21:53:15.570764 I | mvcc: store.index: compact 51461225
>2018-09-25 21:53:15.575587 I | mvcc: finished scheduled compaction at 51461225 (took 3.213734ms)
>2018-09-25 21:54:11.516958 W | etcdserver: apply entries took too long [715.50321ms for 1 entries]
>2018-09-25 21:54:11.516996 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 21:54:12.042866 W | wal: sync duration of 1.140695339s, expected less than 1s
>2018-09-25 21:54:12.042931 W | etcdserver: failed to send out heartbeat on time (exceeded the 500ms timeout for 202.137476ms)
>2018-09-25 21:54:12.042937 W | etcdserver: server is likely overloaded
>2018-09-25 21:54:12.042946 W | etcdserver: failed to send out heartbeat on time (exceeded the 500ms timeout for 202.155295ms)
>2018-09-25 21:54:12.042960 W | etcdserver: server is likely overloaded
>2018-09-25 21:54:15.077183 I | raft: b86402c9b4aca9cc [logterm: 525, index: 218848585, vote: b86402c9b4aca9cc] ignored MsgVote from f853b05503ea4e1f [logterm: 525, index: 218848568] at term 525: lease is not expired (remaining ticks: 5)
>2018-09-25 21:54:15.083540 I | raft: b86402c9b4aca9cc [term: 525] received a MsgApp message with higher term from f853b05503ea4e1f [term: 526]
>2018-09-25 21:54:15.083578 I | raft: b86402c9b4aca9cc became follower at term 526
>2018-09-25 21:54:15.083587 I | raft: found conflict at index 218848569 [existing term: 525, conflicting term: 526]
>2018-09-25 21:54:15.083591 I | raft: truncate the unstable entries before index 218848569
>2018-09-25 21:54:15.083599 I | raft: raft.node: b86402c9b4aca9cc changed leader from b86402c9b4aca9cc to f853b05503ea4e1f at term 526
>2018-09-25 21:54:15.803016 W | wal: sync duration of 3.724760072s, expected less than 1s
>2018-09-25 21:54:16.009185 W | etcdserver: apply entries took too long [3.966119692s for 1 entries]
>2018-09-25 21:54:16.009208 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 21:54:23.360437 W | etcdserver: failed to revoke 4e1f66121e7e1e84 ("etcdserver: request timed out")
>2018-09-25 21:54:23.861083 W | etcdserver: failed to revoke 4e1f66121e7e1e84 ("etcdserver: request timed out")
>2018-09-25 21:54:24.364892 W | etcdserver: failed to revoke 4e1f66121e7e1e84 ("etcdserver: request timed out")
>2018-09-25 21:54:24.862275 W | etcdserver: failed to revoke 4e1f66121e7e1e84 ("etcdserver: request timed out")
>2018-09-25 21:58:15.644920 I | mvcc: store.index: compact 51461810
>2018-09-25 21:58:15.650061 I | mvcc: finished scheduled compaction at 51461810 (took 3.132836ms)
>2018-09-25 22:01:04.679330 W | etcdserver: apply entries took too long [115.580773ms for 1 entries]
>2018-09-25 22:01:04.679368 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 22:03:15.675872 I | mvcc: store.index: compact 51462402
>2018-09-25 22:03:15.679839 I | mvcc: finished scheduled compaction at 51462402 (took 2.366093ms)
>2018-09-25 22:08:15.706274 I | mvcc: store.index: compact 51463008
>2018-09-25 22:08:15.710490 I | mvcc: finished scheduled compaction at 51463008 (took 2.269019ms)
>2018-09-25 22:09:38.046770 W | etcdserver: apply entries took too long [134.777452ms for 1 entries]
>2018-09-25 22:09:38.046793 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 22:09:49.817878 W | etcdserver: apply entries took too long [105.136497ms for 1 entries]
>2018-09-25 22:09:49.817907 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 22:13:15.757510 I | mvcc: store.index: compact 51463575
>2018-09-25 22:13:15.761695 I | mvcc: finished scheduled compaction at 51463575 (took 2.17053ms)
>2018-09-25 22:14:11.338021 W | etcdserver: apply entries took too long [168.676783ms for 1 entries]
>2018-09-25 22:14:11.338040 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 22:14:28.089638 W | etcdserver: apply entries took too long [104.077586ms for 1 entries]
>2018-09-25 22:14:28.089662 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 22:17:32.295113 W | etcdserver: apply entries took too long [113.903129ms for 1 entries]
>2018-09-25 22:17:32.295132 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 22:18:15.773484 I | mvcc: store.index: compact 51464130
>2018-09-25 22:18:15.776969 I | mvcc: finished scheduled compaction at 51464130 (took 1.974576ms)
>2018-09-25 22:18:22.325895 W | etcdserver: apply entries took too long [100.600212ms for 1 entries]
>2018-09-25 22:18:22.325912 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 22:19:18.353050 W | etcdserver: apply entries took too long [127.739744ms for 1 entries]
>2018-09-25 22:19:18.353070 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 22:23:15.806059 I | mvcc: store.index: compact 51464690
>2018-09-25 22:23:15.809989 I | mvcc: finished scheduled compaction at 51464690 (took 2.09726ms)
>2018-09-25 22:26:23.238129 W | etcdserver: apply entries took too long [129.230493ms for 1 entries]
>2018-09-25 22:26:23.238154 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 22:27:34.583631 W | etcdserver: apply entries took too long [106.329529ms for 1 entries]
>2018-09-25 22:27:34.583696 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 22:28:15.830503 I | mvcc: store.index: compact 51465285
>2018-09-25 22:28:15.835835 I | mvcc: finished scheduled compaction at 51465285 (took 2.651537ms)
>2018-09-25 22:32:36.209999 W | etcdserver: apply entries took too long [131.908681ms for 1 entries]
>2018-09-25 22:32:36.210053 W | etcdserver: avoid queries with large range/delete range!
>2018-09-25 22:33:15.870817 I | mvcc: store.index: compact 51465894
>2018-09-25 22:33:15.875314 I | mvcc: finished scheduled compaction at 51465894 (took 2.563614ms)
>2018-09-25 22:33:39.558386 W | wal: sync duration of 1.167461834s, expected less than 1s
>2018-09-25 22:34:09.156289 W | wal: sync duration of 1.065266727s, expected less than 1s
>2018-09-25 22:34:40.115889 W | wal: sync duration of 2.11287587s, expected less than 1s
>2018-09-25 22:35:09.354266 W | wal: sync duration of 1.236452385s, expected less than 1s

Actions: View

Attachments on bug 1624493: 1483121 | 1483373 | 1483374 | 1483375 | 1483389 | 1483390 | 1487375 | 1487376 | 1487377