Bug 1622106
| Summary: | docker credentials from host are not available to containerized kubelet in 'oc cluster up' | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Xingxing Xia <xxia> | ||||
| Component: | Node | Assignee: | Vikas Choudhary <vichoudh> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Xingxing Xia <xxia> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 3.11.0 | CC: | aos-bugs, bparees, deads, jokerman, mfojtik, mmccomas, sjenning, vichoudh, wzheng, xtian, xxia | ||||
| Target Milestone: | --- | Flags: | xxia:
needinfo-
|
||||
| Target Release: | 3.11.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-12-21 15:23:34 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Comment 6
Ben Parees
2018-08-24 15:01:10 UTC
David, I put in a fix for some of this behavior here: https://github.com/openshift/origin/pull/20637 but it looks like maybe hyperkube isn't leveraging it? Do we need to mount the user's .docker/config.json into the hyperkube container so it can pull necessary images? (or do something else to ensure it uses the right credentials?) Any other containers we're going to have to do this for? Sending to pod/node team after discussion with David, the node needs to use the private registry credentials (if any) when pulling images. See: https://github.com/openshift/origin/pull/20637 for changes that were made to cluster up to pick up credentials from the host. Failed with below version: oc v3.11.0-0.28.0 kubernetes v1.11.0+d4cacc0 features: Basic-Auth GSSAPI Kerberos SPNEGO Here is failure message, I will attach detailed docker log in attachment. I0905 23:06:14.362933 20797 run.go:200] Container created with id "261bbfe411611f2bcf0e51426544de7adc01af3621263e113a7e87fc2284692b" I0905 23:06:17.416038 20797 run_self_hosted.go:155] started kubelet in container "261bbfe411611f2bcf0e51426544de7adc01af3621263e113a7e87fc2284692b" I0905 23:06:17.416685 20797 loader.go:359] Config loaded from file /root/openshift.local.clusterup/kube-apiserver/admin.kubeconfig I0905 23:06:17.416830 20797 run_self_hosted.go:180] Waiting for the kube-apiserver to be ready ... I0905 23:06:17.417805 20797 round_trippers.go:405] GET https://127.0.0.1:8443/healthz?timeout=32s in 0 milliseconds I0905 23:06:17.417848 20797 run_self_hosted.go:545] Server isn't healthy yet. Waiting a little while. Get https://127.0.0.1:8443/healthz?timeout=32s: dial tcp 127.0.0.1:8443: connect: connection refused I0905 23:06:18.418613 20797 round_trippers.go:405] GET https://127.0.0.1:8443/healthz?timeout=32s in 0 milliseconds I0905 23:06:18.418673 20797 run_self_hosted.go:545] Server isn't healthy yet. Waiting a little while. Get https://127.0.0.1:8443/healthz?timeout=32s: dial tcp 127.0.0.1:8443: connect: connection refused Created attachment 1481190 [details]
Log from ose-node container
Vikas, can you update this? It is a blocker currently. Its working for me :)
[root@mynode ose]# _output/local/bin/linux/amd64/oc cluster up -- --loglevel 6
Getting a Docker client ...
Checking if image openshift/origin-control-plane:v3.11 is available ...
Checking type of volume mount ...
Determining server IP ...
Checking if OpenShift is already running ...
Checking for supported Docker version (=>1.22) ...
Checking if insecured registry is configured properly in Docker ...
Checking if required ports are available ...
Checking if OpenShift client is configured properly ...
Checking if image openshift/origin-control-plane:v3.11 is available ...
Starting OpenShift using openshift/origin-control-plane:v3.11 ...
I0913 11:21:10.756288 94340 config.go:40] Running "create-master-config"
I0913 11:21:12.394768 94340 config.go:46] Running "create-node-config"
I0913 11:21:13.192389 94340 flags.go:30] Running "create-kubelet-flags"
I0913 11:21:13.565973 94340 run_self_hosted.go:497] docker config path /root/.docker/config.json
I0913 11:21:13.566009 94340 run_kubelet.go:49] Running "start-kubelet"
I0913 11:21:13.734037 94340 run_self_hosted.go:180] Waiting for the kube-apiserver to be ready ...
I0913 11:21:46.736444 94340 interface.go:26] Installing "kube-proxy" ...
I0913 11:21:46.736476 94340 interface.go:26] Installing "kube-dns" ...
I0913 11:21:46.736485 94340 interface.go:26] Installing "openshift-service-cert-signer-operator" ...
I0913 11:21:46.736493 94340 interface.go:26] Installing "openshift-apiserver" ...
I0913 11:21:46.736540 94340 apply_template.go:81] Installing "openshift-apiserver"
I0913 11:21:46.736544 94340 apply_template.go:81] Installing "kube-proxy"
I0913 11:21:46.736575 94340 apply_template.go:81] Installing "openshift-service-cert-signer-operator"
I0913 11:21:46.736544 94340 apply_template.go:81] Installing "kube-dns"
I0913 11:21:47.893674 94340 interface.go:41] Finished installing "kube-proxy" "kube-dns" "openshift-service-cert-signer-operator" "openshift-apiserver"
I0913 11:24:28.915341 94340 run_self_hosted.go:232] openshift-apiserver available
I0913 11:24:28.915375 94340 interface.go:26] Installing "openshift-controller-manager" ...
I0913 11:24:28.915395 94340 apply_template.go:81] Installing "openshift-controller-manager"
I0913 11:24:30.449773 94340 interface.go:41] Finished installing "openshift-controller-manager"
Adding default OAuthClient redirect URIs ...
Adding sample-templates ...
Adding persistent-volumes ...
Adding registry ...
Adding router ...
Adding web-console ...
Adding centos-imagestreams ...
I0913 11:24:30.464776 94340 interface.go:26] Installing "sample-templates" ...
I0913 11:24:30.464790 94340 interface.go:26] Installing "persistent-volumes" ...
I0913 11:24:30.464798 94340 interface.go:26] Installing "openshift-image-registry" ...
I0913 11:24:30.464805 94340 interface.go:26] Installing "openshift-router" ...
I0913 11:24:30.464814 94340 interface.go:26] Installing "openshift-web-console-operator" ...
I0913 11:24:30.464823 94340 interface.go:26] Installing "centos-imagestreams" ...
I0913 11:24:30.464875 94340 apply_list.go:67] Installing "centos-imagestreams"
I0913 11:24:30.464896 94340 interface.go:26] Installing "sample-templates/cakephp quickstart" ...
I0913 11:24:30.464913 94340 interface.go:26] Installing "sample-templates/django quickstart" ...
I0913 11:24:30.464921 94340 interface.go:26] Installing "sample-templates/jenkins pipeline ephemeral" ...
I0913 11:24:30.464928 94340 interface.go:26] Installing "sample-templates/mongodb" ...
I0913 11:24:30.464935 94340 interface.go:26] Installing "sample-templates/mysql" ...
I0913 11:24:30.464941 94340 interface.go:26] Installing "sample-templates/postgresql" ...
I0913 11:24:30.464951 94340 interface.go:26] Installing "sample-templates/dancer quickstart" ...
I0913 11:24:30.464958 94340 interface.go:26] Installing "sample-templates/nodejs quickstart" ...
I0913 11:24:30.464966 94340 interface.go:26] Installing "sample-templates/rails quickstart" ...
I0913 11:24:30.464973 94340 interface.go:26] Installing "sample-templates/sample pipeline" ...
I0913 11:24:30.464979 94340 interface.go:26] Installing "sample-templates/mariadb" ...
I0913 11:24:30.464988 94340 apply_list.go:67] Installing "sample-templates/jenkins pipeline ephemeral"
I0913 11:24:30.465012 94340 apply_list.go:67] Installing "sample-templates/mysql"
I0913 11:24:30.465018 94340 apply_list.go:67] Installing "sample-templates/dancer quickstart"
I0913 11:24:30.465027 94340 apply_list.go:67] Installing "sample-templates/mariadb"
I0913 11:24:30.465023 94340 apply_list.go:67] Installing "sample-templates/mongodb"
I0913 11:24:30.465033 94340 apply_list.go:67] Installing "sample-templates/cakephp quickstart"
I0913 11:24:30.465044 94340 apply_list.go:67] Installing "sample-templates/django quickstart"
I0913 11:24:30.465012 94340 apply_list.go:67] Installing "sample-templates/postgresql"
I0913 11:24:30.465147 94340 apply_list.go:67] Installing "sample-templates/rails quickstart"
I0913 11:24:30.465182 94340 apply_list.go:67] Installing "sample-templates/sample pipeline"
I0913 11:24:30.465204 94340 apply_list.go:67] Installing "sample-templates/nodejs quickstart"
I0913 11:24:30.465578 94340 apply_template.go:81] Installing "openshift-web-console-operator"
I0913 11:24:31.806844 94340 interface.go:41] Finished installing "sample-templates/cakephp quickstart" "sample-templates/django quickstart" "sample-templates/jenkins pipeline ephemeral" "sample-templates/mongodb" "sample-templates/mysql" "sample-templates/postgresql" "sample-templates/dancer quickstart" "sample-templates/nodejs quickstart" "sample-templates/rails quickstart" "sample-templates/sample pipeline" "sample-templates/mariadb"
I0913 11:24:57.239988 94340 interface.go:41] Finished installing "sample-templates" "persistent-volumes" "openshift-image-registry" "openshift-router" "openshift-web-console-operator" "centos-imagestreams"
Login to server ...
Creating initial project "myproject" ...
Server Information ...
OpenShift server started.
The server is accessible via web console at:
https://127.0.0.1:8443
You are logged in as:
User: developer
Password: <any value>
To login as administrator:
oc login -u system:admin
[root@mynode ose]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/openshift/origin-node v3.11 f9162d43f7fb 11 hours ago 1.16 GB
docker.io/openshift/origin-control-plane v3.11 09a6283cb346 11 hours ago 820 MB
docker.io/openshift/origin-haproxy-router v3.11 d4d38b7e26f7 11 hours ago 403 MB
docker.io/openshift/origin-hyperkube v3.11 6cbab4455ce5 11 hours ago 503 MB
docker.io/openshift/origin-deployer v3.11 f76413ba883e 11 hours ago 378 MB
docker.io/openshift/origin-hypershift v3.11 7da7dda99b09 11 hours ago 544 MB
docker.io/openshift/origin-cli v3.11 a980d432d396 11 hours ago 378 MB
docker.io/openshift/origin-pod v3.11 6788b0f14208 11 hours ago 256 MB
docker.io/openshift/origin-service-serving-cert-signer v3.11 0320a92eb388 6 days ago 279 MB
docker.io/openshift/origin-web-console v3.11 d6c6e1f1e643 7 days ago 341 MB
docker.io/openshift/origin-docker-registry v3.11 84d4c3423ddc 7 days ago 307 MB
vikaschoudhary16/sample-device-plugin-amd64 1.0 0f180fc29825 2 weeks ago 228 MB
docker.io/vikaschoudhary16/kubernetes-e2e-test-images/sample-device-plugin-amd64 1.0 0f180fc29825 2 weeks ago 228 MB
docker.io/vikaschoudhary16/sample-device-plugin-amd64 1.0 0f180fc29825 2 weeks ago 228 MB
gcr.io/kubernetes-e2e-test-images/sample-device-plugin-amd64 1.0 0f180fc29825 2 weeks ago 228 MB
vikaschoudhary16/kubernetes-e2e-test-images/sample-device-plugin-amd64 1.0 0f180fc29825 2 weeks ago 228 MB
<none> <none> 0b3e1c9861f4 2 weeks ago 228 MB
gcr.io/kubernetes-e2e-test-images/webhook-ppc64le 1.12v2 787657a0f38f 2 weeks ago 18.7 MB
gcr.io/kubernetes-e2e-test-images/webhook-arm64 1.12v2 fc622ae9604e 2 weeks ago 17.3 MB
gcr.io/kubernetes-e2e-test-images/webhook-arm 1.12v2 1e8269d38e9c 2 weeks ago 15.8 MB
gcr.io/kubernetes-e2e-test-images/webhook-amd64 1.12v2 427e658923a8 2 weeks ago 18.3 MB
docker.io/ubuntu latest 16508e5c265d 3 weeks ago 84.1 MB
gcr.io/kubernetes-e2e-test-images/vtest latest 16508e5c265d 3 weeks ago 84.1 MB
gcr.io/openshift-gce-devel/vtest latest 16508e5c265d 3 weeks ago 84.1 MB
test latest 16508e5c265d 3 weeks ago 84.1 MB
gcr.io/openshift-gce-devel/kubemark k3tszu 39db22788b95 3 weeks ago 235 MB
gcr.io/openshift-gce-devel/kubemark jw00x3 265cecb61969 3 weeks ago 235 MB
gcr.io/openshift-gce-devel/kubemark o4atfr 265cecb61969 3 weeks ago 235 MB
gcr.io/openshift-gce-devel/kubemark sntik3 265cecb61969 3 weeks ago 235 MB
gcr.io/openshift-gce-devel/kubemark tbd6vh 7b75038e087a 3 weeks ago 235 MB
gcr.io/openshift-gce-devel/kubemark i9bufk a2318b38fa60 3 weeks ago 235 MB
gcr.io/openshift-gce-devel/kubemark jtwa4d a2318b38fa60 3 weeks ago 235 MB
docker.io/centos 7 5182e96772bf 5 weeks ago 200 MB
<none> <none> e7728a35611a 6 weeks ago 2.58 GB
docker.io/busybox latest e1ddd7948a1c 6 weeks ago 1.16 MB
<none> <none> e7798f6b339c 7 weeks ago 2.58 GB
docker.io/golang 1.10.3 d0e7a411e3da 8 weeks ago 794 MB
docker.io/debian jessie 79f4bda91989 8 weeks ago 127 MB
docker.io/alpine 3.6 da579b235e92 2 months ago 4.03 MB
k8s.gcr.io/kube-cross v1.10.3-1 aeec97de219a 2 months ago 2.08 GB
k8s.gcr.io/debian-hyperkube-base-amd64 0.10 7812d248bfc9 5 months ago 398 MB
k8s.gcr.io/debian-iptables-amd64 v10 196b441f5192 7 months ago 45.7 MB
k8s.gcr.io/pause 3.1 da86e6ba6ca1 8 months ago 742 kB
docker.io/arm64v8/alpine 3.6 a9277cb286bb 10 months ago 3.78 MB
docker.io/ppc64le/alpine 3.6 daa414b19dcf 10 months ago 4.72 MB
docker.io/arm32v6/alpine 3.6 8ecf96c9b538 10 months ago 3.62 MB
[root@mynode ose]# _output/local/bin/linux/amd64/oc version
oc v3.11.0-0.28.0+30d224c
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO
Server https://127.0.0.1:8443
kubernetes v1.11.0+d4cacc0
Did you do `docker login` to private registry first?
please make sure that you are able to pull image using `docker pull`. Also please share the terminal output similar to what i shared above. Verified.
oc cluster down
umount `mount | grep -o '[^ ]*openshift.*volumes[^ ]*'`
rm -rf openshift.local.clusterup
docker rmi `docker images -q`
docker images # no images
oc cluster up --image='registry.dev.redhat.io/openshift3/ose-${component}:${version}'
Get:
...
Pulling image registry.dev.redhat.io/openshift3/ose-control-plane:v3.11
...
I0913 22:37:45.252058 1690 apply_template.go:81] Installing "openshift-web-console-operator"
...
The server is accessible via web console at:
https://127.0.0.1:8443
You are logged in as:
User: developer
Password: <any value>
To login as administrator:
oc login -u system:admin
# oc get po --all-namespaces
All are running
But *-service-serving-cert-signer is pulled from docker.io and named ose-*. I may file a separate bug for it. docker images REPOSITORY TAG IMAGE ID CREATED SIZE registry.dev.redhat.io/openshift3/ose-node v3.11 492d5e43e0a4 4 hours ago 1.16 GB registry.dev.redhat.io/openshift3/ose-control-plane v3.11 3b0994119342 4 hours ago 802 MB registry.dev.redhat.io/openshift3/ose-haproxy-router v3.11 6d51cd31bc5c 4 hours ago 374 MB registry.dev.redhat.io/openshift3/ose-deployer v3.11 a65ecb778800 4 hours ago 357 MB registry.dev.redhat.io/openshift3/ose-cli v3.11 59a8c96f6c30 4 hours ago 357 MB registry.dev.redhat.io/openshift3/ose-hypershift v3.11 f840e5b056ff 4 hours ago 522 MB registry.dev.redhat.io/openshift3/ose-web-console v3.11 80508b47d1c4 4 hours ago 318 MB registry.dev.redhat.io/openshift3/ose-hyperkube v3.11 b00212def036 4 hours ago 482 MB registry.dev.redhat.io/openshift3/ose-docker-registry v3.11 b957e1d2b220 4 hours ago 284 MB registry.dev.redhat.io/openshift3/ose-pod v3.11 8f946ab55a0c 4 hours ago 234 MB docker.io/openshift/origin-service-serving-cert-signer v3.11 0320a92eb388 7 days ago 279 MB Closing bugs that were verified and targeted for GA but for some reason were not picked up by errata. This bug fix should be present in current 3.11 release content. |