Bug 1873590 - [4.6] rhcos nightly builds contain unstable etcd
Summary: [4.6] rhcos nightly builds contain unstable etcd
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Release
Version: 4.6
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.6.0
Assignee: Justin Pierce
QA Contact: ge liu
: 1868025 1872598 (view as bug list)
Depends On:
TreeView+ depends on / blocked
Reported: 2020-08-28 17:19 UTC by Sam Batschelet
Modified: 2020-10-27 16:35 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2020-10-27 16:35:38 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:35:55 UTC

Description Sam Batschelet 2020-08-28 17:19:47 UTC
Description of problem: Description of problem: The build process for rhcos had divergence in the golang versions that were used to compile etcd. The CI images were compiled with golang 1.12. While the rhcos builds were compiled using golang 1.14.

THe problem with that is bbolt(KVS) version used in 3.4.9 is not compatible with golang 1.14. Although the binary compiles, unit tests panic[1].

It appears that the issue can manifest itself with very high etcd CPU utilization that can result in OOM. Other issues may also exist. 

For this reason any cluster built with these images should be considered tainted. Until such time that rhcos builds contain the correct etcd binary image. CI nightly images would perhaps be the next choice.

[1] https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_release/11302/rehearse-11302-pull-ci-openshift-etcd-openshift-4.6-unit/1298751792531640320

Version-Release number of selected component (if applicable):

How reproducible: unclear, assumption is 100%

Steps to Reproduce:
1. oc logs -n openshift-etcd $POD -c etcd | grep 'Go Version:'

Actual results: etcd is compiled with golang 1.14

Expected results: etcd is compiled with golang 1.12

Additional info:

Comment 2 Sam Batschelet 2020-08-28 17:26:36 UTC
*** Bug 1873412 has been marked as a duplicate of this bug. ***

Comment 3 Sam Batschelet 2020-08-28 17:32:49 UTC
*** Bug 1872598 has been marked as a duplicate of this bug. ***

Comment 4 Colin Walters 2020-08-28 17:46:06 UTC
Also RHCOS doesn't include etcd, it comes as a container image right?  So Build seems more likely yes.

Comment 5 Justin Pierce 2020-08-28 19:11:33 UTC
ART / production builds changed to golang 1.12 for etcd on 8/26 with this commit: https://github.com/openshift/ocp-build-data/commit/2c8eb9dd53cf69b010c58b12837a93f3b3ac4a8f

Confirmed in build logs that etcd is being compiled with 1.12 in recent nightlies with timestamps >= 20200826.195121.

From the latest nightly:
[jupierce@localhost Downloads]$ oc adm release info registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-08-28-102309 --pullspecs | grep etcd
  cluster-etcd-operator                          quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7673f35971fac8fe36eddd732a4ef62469bdbb7db22a777c90524f888889b5fa
  etcd                                           quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:52d6a7a9e653589e4ba8a21c189d99ec31c265c310b5f0f14949536366cb4d84
[jupierce@localhost Downloads]$ docker run -it --rm quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:52d6a7a9e653589e4ba8a21c189d99ec31c265c310b5f0f14949536366cb4d84
Unable to find image 'quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:52d6a7a9e653589e4ba8a21c189d99ec31c265c310b5f0f14949536366cb4d84' locally
sha256:52d6a7a9e653589e4ba8a21c189d99ec31c265c310b5f0f14949536366cb4d84: Pulling from openshift-release-dev/ocp-v4.0-art-dev
c9fa7d57b902: Already exists 
74cbb6607642: Already exists 
c676df4ac84e: Already exists 
ad956945835b: Already exists 
0fc958457837: Pull complete 
Digest: sha256:52d6a7a9e653589e4ba8a21c189d99ec31c265c310b5f0f14949536366cb4d84
Status: Downloaded newer image for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:52d6a7a9e653589e4ba8a21c189d99ec31c265c310b5f0f14949536366cb4d84
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2020-08-28 19:08:57.824441 I | etcdmain: etcd Version: 3.4.9
2020-08-28 19:08:57.825333 I | etcdmain: Git SHA: f3fdb32
2020-08-28 19:08:57.825341 I | etcdmain: Go Version: go1.12.12     <== HERE

Nightlies are not presently passing CI, however, due to other disruptions caused during the migration.

Comment 9 Stefan Schimanski 2020-08-31 07:01:48 UTC
*** Bug 1868025 has been marked as a duplicate of this bug. ***

Comment 10 Mike Gahagan 2020-08-31 20:51:04 UTC
I can confirm 4.6.0-0.nightly-2020-08-27-005538 has an etcd that was built with 1.12.12

[m@localhost mgahagan-133108]$ oc logs -n openshift-etcd etcd-mgahagan-133108-x98t6-master-0 -c etcd | grep 'Go Version:'
2020-08-31 17:19:46.803283 I | etcdmain: Go Version: go1.12.12
[m@localhost mgahagan-133108]$ oc logs -n openshift-etcd etcd-mgahagan-133108-x98t6-master-1 -c etcd | grep 'Go Version:'
2020-08-31 17:21:03.646744 I | etcdmain: Go Version: go1.12.12
[m@localhost mgahagan-133108]$ oc logs -n openshift-etcd etcd-mgahagan-133108-x98t6-master-2 -c etcd | grep 'Go Version:'
2020-08-31 17:19:03.160948 I | etcdmain: Go Version: go1.12.12

Comment 11 ge liu 2020-09-21 12:24:38 UTC
Yes, verified 4.6.0-0.nightly-2020-09-20-184226

2020-09-21 12:23:17.560850 I | etcdmain: etcd Version: 3.4.9
2020-09-21 12:23:17.560856 I | etcdmain: Git SHA: 4aa8f02
2020-09-21 12:23:17.560860 I | etcdmain: Go Version: go1.12.12
2020-09-21 12:23:17.560870 I | etcdmain: Go OS/Arch: linux/amd64

Comment 13 errata-xmlrpc 2020-10-27 16:35:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.