Description of problem: kubectl exec fails like this on Fedora 23: # kubectl exec mock-ooycy /bin/sh -i error: Unable to upgrade connection: { "kind": "Status", "apiVersion": "v1", "metadata": {}, "status": "Failure", "message": "tls: handshake did not verify certificate chain", "code": 500 } Version-Release number of selected component (if applicable): kubernetes-1.1.0-0.16.git388061f.fc23.x86_64 How reproducible: Every time Steps to Reproduce: 1. See above
I am able to reproduce it. Steps: 1) install the latest k8s and start all services 2) update apiserver config and remove ServiceAccount from admission control 3) restart kube-apiserver 3) create mock.yaml file:# cat mock.yaml apiVersion: v1 kind: Pod metadata: name: mock spec: containers: - name: mock image: rusintez/mock 4) kubectl create -f mock.yaml Running command: # kubectl exec --v=9 mock /bin/sh -i I1027 09:11:04.821317 9945 debugging.go:101] curl -k -v -XGET -H "User-Agent: kubectl/v1.0.6 (linux/amd64) kubernetes/388061f" http://localhost:8080/api I1027 09:11:04.822881 9945 debugging.go:120] GET http://localhost:8080/api 200 OK in 1 milliseconds I1027 09:11:04.822904 9945 debugging.go:126] Response Headers: I1027 09:11:04.822915 9945 debugging.go:129] Content-Length: 32 I1027 09:11:04.822927 9945 debugging.go:129] Content-Type: application/json I1027 09:11:04.822937 9945 debugging.go:129] Date: Tue, 27 Oct 2015 08:11:04 GMT I1027 09:11:04.822968 9945 request.go:779] Response Body: { "versions": [ "v1" ] } I1027 09:11:04.823272 9945 debugging.go:101] curl -k -v -XGET -H "User-Agent: kubectl/v1.0.6 (linux/amd64) kubernetes/388061f" http://localhost:8080/api/v1/namespaces/default/pods/mock I1027 09:11:04.826041 9945 debugging.go:120] GET http://localhost:8080/api/v1/namespaces/default/pods/mock 200 OK in 2 milliseconds I1027 09:11:04.826065 9945 debugging.go:126] Response Headers: I1027 09:11:04.826077 9945 debugging.go:129] Content-Type: application/json I1027 09:11:04.826087 9945 debugging.go:129] Date: Tue, 27 Oct 2015 08:11:04 GMT I1027 09:11:04.826098 9945 debugging.go:129] Content-Length: 973 I1027 09:11:04.826143 9945 request.go:779] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"mock","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/mock","uid":"33334b8c-7c81-11e5-a55a-525400e11373","resourceVersion":"2196","creationTimestamp":"2015-10-27T08:03:25Z"},"spec":{"containers":[{"name":"mock","image":"rusintez/mock","resources":{},"terminationMessagePath":"/dev/termination-log","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","dnsPolicy":"ClusterFirst","nodeName":"127.0.0.1"},"status":{"phase":"Running","conditions":[{"type":"Ready","status":"True"}],"hostIP":"127.0.0.1","podIP":"172.17.0.1","startTime":"2015-10-27T08:03:25Z","containerStatuses":[{"name":"mock","state":{"running":{"startedAt":"2015-10-27T08:05:41Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"rusintez/mock","imageID":"docker://ca5d0df8796cc6bc30808508084d3ed31b229aec6678ceaf52b5fe295adacac9","containerID":"docker://4f96a3af5476d8cd5d80269976dfa0defb4081fd5b57cc5d2bdcc3704985d951"}]}} I1027 09:11:04.826794 9945 exec.go:127] defaulting container name to mock F1027 09:11:04.861602 9945 helpers.go:72] error: Unable to upgrade connection: { "kind": "Status", "apiVersion": "v1", "metadata": {}, "status": "Failure", "message": "tls: handshake did not verify certificate chain", "code": 500 }
The same happends with different pod: # cat ubuntu.yaml apiVersion: v1 kind: Pod metadata: name: counter spec: containers: - name: count image: ubuntu:14.04 # kubectl exec counter /bin/sh error: Unable to upgrade connection: { "kind": "Status", "apiVersion": "v1", "metadata": {}, "status": "Failure", "message": "tls: handshake did not verify certificate chain", "code": 500 }
Based on [1] I have tried to run the scenario with f22 builds (build with golang-1.4.2-3.fc22.x86_64). # kubectl exec counter /bin/sh -i error: Error executing remote command: Error executing command in container: container not found ("count") tls error is gone. [1] http://stackoverflow.com/questions/32449316/kubernetes-on-mesos-tls-error-on-kubectl-exec
When running the following: # cat counter.yaml apiVersion: v1 kind: Pod metadata: name: counter spec: containers: - name: count image: ubuntu:14.04 args: [bash, -c, 'for ((i = 0; ; i++)); do echo "$i: $(date)"; sleep 1; done'] # kubectl exec counter /bin/bash -i uname Linux ^C^C Confirming exec command is working is supposed to. This is another issue with go-1.5 compiler. I can try to rebuilt f23 with the latest golang-1.5 which is golang-1.5.1-1.fc23. The f23 kubernetes is built with golang-1.5-8.fc23.x86_64
k8s built with golang-1.5.1-1.fc23 suffers the same issue.
Jakub, Vincent, what is the current state of golang in f23? How are we doing with TLS support?
The issue is fixed in 1.2.0 version of kubernetes.
backported code from 1.2.0: http://pkgs.fedoraproject.org/cgit/kubernetes.git/commit/?h=f23&id=b082122255bca3efa706bfdeef7c26ef52bc7578
kubernetes-1.1.0-0.17.git388061f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-c7fa818f3f
kubernetes-1.1.0-0.17.git388061f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. If you want to test the update, you can install it with $ su -c 'dnf --enablerepo=updates-testing update kubernetes' You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-c7fa818f3f
kubernetes-1.1.0-0.17.git388061f.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.