Description of problem: Description of problem: The build process for rhcos had divergence in the golang versions that were used to compile etcd. The CI images were compiled with golang 1.12. While the rhcos builds were compiled using golang 1.14.
THe problem with that is bbolt(KVS) version used in 3.4.9 is not compatible with golang 1.14. Although the binary compiles, unit tests panic.
It appears that the issue can manifest itself with very high etcd CPU utilization that can result in OOM. Other issues may also exist.
For this reason any cluster built with these images should be considered tainted. Until such time that rhcos builds contain the correct etcd binary image. CI nightly images would perhaps be the next choice.
Version-Release number of selected component (if applicable):
How reproducible: unclear, assumption is 100%
Steps to Reproduce:
1. oc logs -n openshift-etcd $POD -c etcd | grep 'Go Version:'
Actual results: etcd is compiled with golang 1.14
Expected results: etcd is compiled with golang 1.12
*** Bug 1873412 has been marked as a duplicate of this bug. ***
*** Bug 1872598 has been marked as a duplicate of this bug. ***
Also RHCOS doesn't include etcd, it comes as a container image right? So Build seems more likely yes.
ART / production builds changed to golang 1.12 for etcd on 8/26 with this commit: https://github.com/openshift/ocp-build-data/commit/2c8eb9dd53cf69b010c58b12837a93f3b3ac4a8f
Confirmed in build logs that etcd is being compiled with 1.12 in recent nightlies with timestamps >= 20200826.195121.
From the latest nightly:
[jupierce@localhost Downloads]$ oc adm release info registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-08-28-102309 --pullspecs | grep etcd
[jupierce@localhost Downloads]$ docker run -it --rm quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:52d6a7a9e653589e4ba8a21c189d99ec31c265c310b5f0f14949536366cb4d84
Unable to find image 'quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:52d6a7a9e653589e4ba8a21c189d99ec31c265c310b5f0f14949536366cb4d84' locally
sha256:52d6a7a9e653589e4ba8a21c189d99ec31c265c310b5f0f14949536366cb4d84: Pulling from openshift-release-dev/ocp-v4.0-art-dev
c9fa7d57b902: Already exists
74cbb6607642: Already exists
c676df4ac84e: Already exists
ad956945835b: Already exists
0fc958457837: Pull complete
Status: Downloaded newer image for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:52d6a7a9e653589e4ba8a21c189d99ec31c265c310b5f0f14949536366cb4d84
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2020-08-28 19:08:57.824441 I | etcdmain: etcd Version: 3.4.9
2020-08-28 19:08:57.825333 I | etcdmain: Git SHA: f3fdb32
2020-08-28 19:08:57.825341 I | etcdmain: Go Version: go1.12.12 <== HERE
Nightlies are not presently passing CI, however, due to other disruptions caused during the migration.
*** Bug 1868025 has been marked as a duplicate of this bug. ***
I can confirm 4.6.0-0.nightly-2020-08-27-005538 has an etcd that was built with 1.12.12
[m@localhost mgahagan-133108]$ oc logs -n openshift-etcd etcd-mgahagan-133108-x98t6-master-0 -c etcd | grep 'Go Version:'
2020-08-31 17:19:46.803283 I | etcdmain: Go Version: go1.12.12
[m@localhost mgahagan-133108]$ oc logs -n openshift-etcd etcd-mgahagan-133108-x98t6-master-1 -c etcd | grep 'Go Version:'
2020-08-31 17:21:03.646744 I | etcdmain: Go Version: go1.12.12
[m@localhost mgahagan-133108]$ oc logs -n openshift-etcd etcd-mgahagan-133108-x98t6-master-2 -c etcd | grep 'Go Version:'
2020-08-31 17:19:03.160948 I | etcdmain: Go Version: go1.12.12
Yes, verified 4.6.0-0.nightly-2020-09-20-184226
2020-09-21 12:23:17.560850 I | etcdmain: etcd Version: 3.4.9
2020-09-21 12:23:17.560856 I | etcdmain: Git SHA: 4aa8f02
2020-09-21 12:23:17.560860 I | etcdmain: Go Version: go1.12.12
2020-09-21 12:23:17.560870 I | etcdmain: Go OS/Arch: linux/amd64
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.