Bug 1819103

Summary: openshift-apiserver pod has lots of spam logs "E0331...etcdserver: mvcc: required revision has been compacted"
Product: OpenShift Container Platform Reporter: Xingxing Xia <xxia>
Component: openshift-apiserverAssignee: Stefan Schimanski <sttts>
Status: CLOSED ERRATA QA Contact: Xingxing Xia <xxia>
Severity: medium Docs Contact:
Priority: high    
Version: 4.4CC: akashem, aos-bugs, ingvarr.zhmakin, mfojtik, mnewby
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-07-13 17:24:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Xingxing Xia 2020-03-31 08:26:07 UTC
Description of problem:
openshift-apiserver pod has lots of spam logs "E0331...etcdserver: mvcc: required revision has been compacted".

There is bug 1537270 found since v3 annoying user. Using this bug for v4 tracker. Not a release blocker but worth addressing in v4.

Version-Release number of selected component (if applicable):
4.4.0-0.nightly-2020-03-29-132004

How reproducible:
Always

Steps to Reproduce:
1. In any normal cluster, check openshift-apiserver logs
$ oc logs -n openshift-apiserver apiserver-684c4c58fd-zhrkk

Actual results:
1. Lots of below spam logs with the high frequency as shown:
...
E0331 07:14:37.992250       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
E0331 07:15:17.454232       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
E0331 07:16:51.789029       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
E0331 07:17:49.103900       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
E0331 07:18:21.208658       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
E0331 07:19:59.232187       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
E0331 07:20:27.581679       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
E0331 07:23:35.898078       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
E0331 07:23:41.608829       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
E0331 07:23:58.527621       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
E0331 07:24:13.190186       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
E0331 07:25:53.514313       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
...

Expected results:
1. No such spam logs. Or no such high frequency.

Additional info:

Comment 1 Maru Newby 2020-04-03 17:49:07 UTC
afaik `mvcc: required revision has been compacted` is a healthy message. Why does it need to be emitted as an error though? The frequency is also concerning.

Comment 2 Abu Kashem 2020-04-06 15:31:04 UTC
We agree, it should be fixed, setting the target release to 4.5. The expectation is:
- This is a healthy message, so we might need to find a way to make it less frequent.
- This should not be labeled as an error message.

Comment 3 Stefan Schimanski 2020-05-06 10:58:46 UTC
Reduced to warning in https://github.com/kubernetes/kubernetes/pull/84602 with 1.18 rebase.

Comment 6 Xingxing Xia 2020-05-11 07:20:57 UTC
Verified in 4.5.0-0.nightly-2020-05-11-032504. Now OAS log shows it as Warning.
$ oc logs -n openshift-apiserver apiserver-79d4f8fd98-9j5bj
...
W0511 07:02:16.669394       1 watcher.go:199] watch chan error: etcdserver: mvcc: required revision has been compacted
W0511 07:02:16.762114       1 watcher.go:199] watch chan error: etcdserver: mvcc: required revision has been compacted
...
W0511 07:11:27.759756       1 watcher.go:199] watch chan error: etcdserver: mvcc: required revision has been compacted
W0511 07:11:36.676603       1 watcher.go:199] watch chan error: etcdserver: mvcc: required revision has been compacted
...

Comment 8 errata-xmlrpc 2020-07-13 17:24:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409