Bug 1831042 - kube-apiserver fails to live reload client CA and front proxy CA
Summary: kube-apiserver fails to live reload client CA and front proxy CA
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-apiserver
Version: 4.4
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.5.0
Assignee: Tomáš Nožička
QA Contact: Xingxing Xia
URL:
Whiteboard:
Depends On:
Blocks: 1831045
TreeView+ depends on / blocked
 
Reported: 2020-05-04 14:12 UTC by Tomáš Nožička
Modified: 2020-07-13 17:34 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1831045 (view as bug list)
Environment:
Last Closed: 2020-07-13 17:34:39 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github kubernetes kubernetes pull 90360 0 None closed Fix client-ca dynamic reload in apiserver 2020-10-19 01:36:12 UTC
Github openshift origin pull 24942 0 None closed Bug 1831042: Fix client-ca dynamic reload in apiserver 2020-10-19 01:36:12 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:34:57 UTC

Description Tomáš Nožička 2020-05-04 14:12:03 UTC
kube-apiserver fails to live reload client CA and front proxy CA

Comment 7 Xingxing Xia 2020-05-12 13:54:22 UTC
Tomáš Nožička, also tried that, just hit other error:
[xxia@pc 2020-05-12 21:31:42 CST all]$ cd /data/src/github.com/openshift/ose
[xxia@pc 2020-05-12 21:31:48 CST ose]$ go test -race -mod=vendor vendor/k8s.io/kubernetes/test/integration/apiserver/certreload/ -run='TestClientCA'
build flag -mod=vendor only valid when using modules
[xxia@pc 2020-05-12 21:32:18 CST ose]$ export GO111MODULE=on
[xxia@pc 2020-05-12 21:32:52 CST ose]$ go test -race -mod=vendor vendor/k8s.io/kubernetes/test/integration/apiserver/certreload/ -run='TestClientCA'
go: cannot find main module, but found Godeps/Godeps.json in /data/src/github.com/openshift/ose
	to create a module there, run:
	go mod init
[xxia@pc 2020-05-12 21:32:53 CST ose]$ go mod init
go: creating new go.mod: module github.com/openshift/origin
go: copying requirements from Godeps/Godeps.json
go: converting Godeps/Godeps.json: stat k8s.io/kubernetes/pkg/api@2f054b7646dc9e98f6dea458d2fb65e1d2c1f731: unknown revision 2f054b7646dc9e98f6dea458d2fb65e1d2c1f731
[xxia@pc 2020-05-12 21:35:52 CST ose]$ echo $?
0
[xxia@pc 2020-05-12 21:35:57 CST ose]$ go test -race -mod=vendor vendor/k8s.io/kubernetes/test/integration/apiserver/certreload/ -run='TestClientCA'
go: inconsistent vendoring in /data/src/github.com/openshift/ose:
	github.com/coreos/etcd@v3.3.10+incompatible: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt

run 'go mod vendor' to sync, or use -mod=mod or -mod=readonly to ignore the vendor directory
[xxia@pc 2020-05-12 21:36:00 CST ose]$ go mod vendor
go: finding module for package k8s.io/component-base/metrics/prometheus/restclient
go: finding module for package github.com/openshift/api/image/v1
...
go: found github.com/go-openapi/jsonreference in github.com/go-openapi/jsonreference v0.19.3
go: github.com/openshift/origin/test/extended/dr imports
	k8s.io/kubernetes/staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured: k8s.io/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20200512103021-70948498bcc6: parsing go.mod:
	module declares its path as: k8s.io/apimachinery
	        but was required as: k8s.io/kubernetes/staging/src/k8s.io/apimachinery
[xxia@pc 2020-05-12 21:39:39 CST ose]$ echo $?
1

# same error as above
[xxia@pc 2020-05-12 21:50:01 CST ose]$ go test -race -mod=vendor vendor/k8s.io/kubernetes/test/integration/apiserver/certreload/ -run='TestClientCA'
go: inconsistent vendoring in /data/src/github.com/openshift/ose:
	github.com/coreos/etcd@v3.3.10+incompatible: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt

run 'go mod vendor' to sync, or use -mod=mod or -mod=readonly to ignore the vendor directory

Could you help guide again? Thanks!

Comment 8 Tomáš Nožička 2020-05-14 07:46:36 UTC
> build flag -mod=vendor only valid when using modules
this implies you have old go as modules are now enabled by default, probably missing some fixes for modules and vendor. OCP 4.5 and 4.4 require golang 1.13. You shouldn't need to init modules as you have vendor dir present - you'd likely get different versions so that's not the way.

the new version give better errors
$ go test -mod=vendor -race ./vendor/k8s.io/kubernetes/test/integration/apiserver/certreload/ -run='TestClientCA' -v
build flag -mod=vendor only valid when using modules

so when you hit it (because it's not using modules yet), just drop the `-mod=vendor` flag (which we use for modules to force vendor dir (default before modules came)
$ go test -race ./vendor/k8s.io/kubernetes/test/integration/apiserver/certreload/ -run='TestClientCA' -v

Apologies for posting the wrong command initially, kube (where this originally landed) and our other repos are on modules. 
(I rather always add `-mod=vendor` initially just to be sure as the go mod logic ignoring vendor dir when present is flawed. There are some fixes targeting that for 1.14 or later I think.)

Comment 9 Xingxing Xia 2020-05-15 04:06:32 UTC
> this implies you have old go ... require golang 1.13
Hmm, as comment 5 shows, my go "1.14.2" is new enough :)
> just drop the `-mod=vendor` flag
Hmm, I in fact tried with and without it, in combination with dropping ./vendor/k8s.io/kubernetes/ or not, hit different other errors :)

But today, I tried, can pass, hooray! :)

# Pasting the whole execution including addressing middle interruption:
[xxia@pc 2020-05-15 11:47:13 CST ose]$ go test -race ./vendor/k8s.io/kubernetes/test/integration/apiserver/certreload/ -run='TestClientCA' -v

Cannot find etcd, cannot run integration tests
Please see https://git.k8s.io/community/contributors/devel/sig-testing/integration-tests.md#install-etcd-dependency for instructions.

You can use 'hack/install-etcd.sh' to install a copy in third_party/.

F0515 11:48:39.615853    8924 etcd.go:170] cannot run integration tests: unable to start etcd: could not find etcd in PATH: exec: "etcd": executable file not found in $PATH
FAIL    github.com/openshift/ose/vendor/k8s.io/kubernetes/test/integration/apiserver/certreload 0.431s
FAIL

[xxia@pc 2020-05-15 11:51:41 CST ose]$ sudo -E oc cp -n openshift-etcd etcd-xxia1-zmvvt-m-0.c.openshift-qe.internal:/bin/etcd /usr/local
/bin/etcd
[xxia@pc 2020-05-15 11:54:10 CST ose]$ ls -l /usr/local/bin/etcd
-rw-r--r--. 1 root root 31118728 May 15 11:52 /usr/local/bin/etcd
[xxia@pc 2020-05-15 11:54:25 CST ose]$ sudo chmod a+x /usr/local/bin/etcd

[xxia@pc 2020-05-15 11:54:28 CST ose]$ go test -race ./vendor/k8s.io/kubernetes/test/integration/apiserver/certreload/ -run='TestClientCA' -v
I0515 11:54:37.722646    9325 etcd.go:99] starting etcd on http://127.0.0.1:40183
I0515 11:54:37.722810    9325 etcd.go:105] storing etcd data in: /tmp/integration_test_etcd_data248597862
...
raft2020/05/15 11:54:37 INFO: 8e9e05c52164694d switched to configuration voters=(10276657743932975437)
2020-05-15 11:54:37.736936 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
=== RUN   TestClientCA
raft2020/05/15 11:54:38 INFO: 8e9e05c52164694d is starting a new election at term 1
...
I0515 11:55:05.891018    9325 controller.go:606] quota admission added evaluator for: endpoints
I0515 11:55:05.904712    9325 controller.go:606] quota admission added evaluator for: endpointslices.discovery.k8s.io
--- PASS: TestClientCA (32.23s)
I0515 11:55:10.056219    9325 controller.go:181] Shutting down kubernetes service endpoint reconciler
PASS
I0515 11:55:10.059552    9325 etcd.go:127] etcd exit status: signal: killed
W0515 11:55:10.068357    9325 genericapiserver.go:651] failed to create event default/kube-system.160f17639b156e61: rpc error: code = Unavailable desc = transport is closing
E0515 11:55:10.070051    9325 controller.go:184] rpc error: code = Unavailable desc = transport is closing
ok      github.com/openshift/ose/vendor/k8s.io/kubernetes/test/integration/apiserver/certreload 32.671s

Verified in 4.5.0-0.nightly-2020-05-15-011814 env. Thanks!

Comment 10 errata-xmlrpc 2020-07-13 17:34:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.