Description of problem: Description of problem: The build process for rhcos had divergence in the golang versions that were used to compile etcd. The CI images were compiled with golang 1.12. While the rhcos builds were compiled using golang 1.14. THe problem with that is bbolt(KVS) version used in 3.4.9 is not compatible with golang 1.14. Although the binary compiles, unit tests panic[1]. It appears that the issue can manifest itself with very high etcd CPU utilization that can result in OOM. Other issues may also exist. For this reason any cluster built with these images should be considered tainted. Until such time that rhcos builds contain the correct etcd binary image. CI nightly images would perhaps be the next choice. [1] https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_release/11302/rehearse-11302-pull-ci-openshift-etcd-openshift-4.6-unit/1298751792531640320 Version-Release number of selected component (if applicable): How reproducible: unclear, assumption is 100% Steps to Reproduce: 1. oc logs -n openshift-etcd $POD -c etcd | grep 'Go Version:' 2. 3. Actual results: etcd is compiled with golang 1.14 Expected results: etcd is compiled with golang 1.12 Additional info:
*** Bug 1873412 has been marked as a duplicate of this bug. ***
*** Bug 1872598 has been marked as a duplicate of this bug. ***
Also RHCOS doesn't include etcd, it comes as a container image right? So Build seems more likely yes.
ART / production builds changed to golang 1.12 for etcd on 8/26 with this commit: https://github.com/openshift/ocp-build-data/commit/2c8eb9dd53cf69b010c58b12837a93f3b3ac4a8f Confirmed in build logs that etcd is being compiled with 1.12 in recent nightlies with timestamps >= 20200826.195121. From the latest nightly: [jupierce@localhost Downloads]$ oc adm release info registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-08-28-102309 --pullspecs | grep etcd cluster-etcd-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7673f35971fac8fe36eddd732a4ef62469bdbb7db22a777c90524f888889b5fa etcd quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:52d6a7a9e653589e4ba8a21c189d99ec31c265c310b5f0f14949536366cb4d84 [jupierce@localhost Downloads]$ docker run -it --rm quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:52d6a7a9e653589e4ba8a21c189d99ec31c265c310b5f0f14949536366cb4d84 Unable to find image 'quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:52d6a7a9e653589e4ba8a21c189d99ec31c265c310b5f0f14949536366cb4d84' locally sha256:52d6a7a9e653589e4ba8a21c189d99ec31c265c310b5f0f14949536366cb4d84: Pulling from openshift-release-dev/ocp-v4.0-art-dev c9fa7d57b902: Already exists 74cbb6607642: Already exists c676df4ac84e: Already exists ad956945835b: Already exists 0fc958457837: Pull complete Digest: sha256:52d6a7a9e653589e4ba8a21c189d99ec31c265c310b5f0f14949536366cb4d84 Status: Downloaded newer image for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:52d6a7a9e653589e4ba8a21c189d99ec31c265c310b5f0f14949536366cb4d84 [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead 2020-08-28 19:08:57.824441 I | etcdmain: etcd Version: 3.4.9 2020-08-28 19:08:57.825333 I | etcdmain: Git SHA: f3fdb32 2020-08-28 19:08:57.825341 I | etcdmain: Go Version: go1.12.12 <== HERE Nightlies are not presently passing CI, however, due to other disruptions caused during the migration.
*** Bug 1868025 has been marked as a duplicate of this bug. ***
I can confirm 4.6.0-0.nightly-2020-08-27-005538 has an etcd that was built with 1.12.12 [m@localhost mgahagan-133108]$ oc logs -n openshift-etcd etcd-mgahagan-133108-x98t6-master-0 -c etcd | grep 'Go Version:' 2020-08-31 17:19:46.803283 I | etcdmain: Go Version: go1.12.12 [m@localhost mgahagan-133108]$ oc logs -n openshift-etcd etcd-mgahagan-133108-x98t6-master-1 -c etcd | grep 'Go Version:' 2020-08-31 17:21:03.646744 I | etcdmain: Go Version: go1.12.12 [m@localhost mgahagan-133108]$ oc logs -n openshift-etcd etcd-mgahagan-133108-x98t6-master-2 -c etcd | grep 'Go Version:' 2020-08-31 17:19:03.160948 I | etcdmain: Go Version: go1.12.12
Yes, verified 4.6.0-0.nightly-2020-09-20-184226 2020-09-21 12:23:17.560850 I | etcdmain: etcd Version: 3.4.9 2020-09-21 12:23:17.560856 I | etcdmain: Git SHA: 4aa8f02 2020-09-21 12:23:17.560860 I | etcdmain: Go Version: go1.12.12 2020-09-21 12:23:17.560870 I | etcdmain: Go OS/Arch: linux/amd64
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196