Bug 1274854 - kubectl exec: handshake did not verify certificate chain
kubectl exec: handshake did not verify certificate chain
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: kubernetes (Show other bugs)
23
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Jan Chaloupka
Fedora Extras Quality Assurance
:
Depends On: 1276255
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-23 12:45 EDT by Stef Walter
Modified: 2015-11-12 00:23 EST (History)
6 users (show)

See Also:
Fixed In Version: kubernetes-1.1.0-0.17.git388061f.fc23
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-12 00:23:01 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Stef Walter 2015-10-23 12:45:25 EDT
Description of problem:

kubectl exec fails like this on Fedora 23:

# kubectl exec mock-ooycy /bin/sh -i
error: Unable to upgrade connection: {
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "tls: handshake did not verify certificate chain",
  "code": 500
}

Version-Release number of selected component (if applicable):

kubernetes-1.1.0-0.16.git388061f.fc23.x86_64

How reproducible:

Every time

Steps to Reproduce:
1. See above
Comment 1 Jan Chaloupka 2015-10-27 04:19:18 EDT
I am able to reproduce it.

Steps:
1) install the latest k8s and start all services
2) update apiserver config and remove ServiceAccount from admission control
3) restart kube-apiserver
3) create mock.yaml file:# cat mock.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: mock
spec:
  containers:
  - name: mock
    image: rusintez/mock

4) kubectl create -f mock.yaml

Running command:
# kubectl exec --v=9 mock /bin/sh -i
I1027 09:11:04.821317    9945 debugging.go:101] curl -k -v -XGET  -H "User-Agent: kubectl/v1.0.6 (linux/amd64) kubernetes/388061f" http://localhost:8080/api
I1027 09:11:04.822881    9945 debugging.go:120] GET http://localhost:8080/api 200 OK in 1 milliseconds
I1027 09:11:04.822904    9945 debugging.go:126] Response Headers:
I1027 09:11:04.822915    9945 debugging.go:129]     Content-Length: 32
I1027 09:11:04.822927    9945 debugging.go:129]     Content-Type: application/json
I1027 09:11:04.822937    9945 debugging.go:129]     Date: Tue, 27 Oct 2015 08:11:04 GMT
I1027 09:11:04.822968    9945 request.go:779] Response Body: {
  "versions": [
    "v1"
  ]
}
I1027 09:11:04.823272    9945 debugging.go:101] curl -k -v -XGET  -H "User-Agent: kubectl/v1.0.6 (linux/amd64) kubernetes/388061f" http://localhost:8080/api/v1/namespaces/default/pods/mock
I1027 09:11:04.826041    9945 debugging.go:120] GET http://localhost:8080/api/v1/namespaces/default/pods/mock 200 OK in 2 milliseconds
I1027 09:11:04.826065    9945 debugging.go:126] Response Headers:
I1027 09:11:04.826077    9945 debugging.go:129]     Content-Type: application/json
I1027 09:11:04.826087    9945 debugging.go:129]     Date: Tue, 27 Oct 2015 08:11:04 GMT
I1027 09:11:04.826098    9945 debugging.go:129]     Content-Length: 973
I1027 09:11:04.826143    9945 request.go:779] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"mock","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/mock","uid":"33334b8c-7c81-11e5-a55a-525400e11373","resourceVersion":"2196","creationTimestamp":"2015-10-27T08:03:25Z"},"spec":{"containers":[{"name":"mock","image":"rusintez/mock","resources":{},"terminationMessagePath":"/dev/termination-log","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","dnsPolicy":"ClusterFirst","nodeName":"127.0.0.1"},"status":{"phase":"Running","conditions":[{"type":"Ready","status":"True"}],"hostIP":"127.0.0.1","podIP":"172.17.0.1","startTime":"2015-10-27T08:03:25Z","containerStatuses":[{"name":"mock","state":{"running":{"startedAt":"2015-10-27T08:05:41Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"rusintez/mock","imageID":"docker://ca5d0df8796cc6bc30808508084d3ed31b229aec6678ceaf52b5fe295adacac9","containerID":"docker://4f96a3af5476d8cd5d80269976dfa0defb4081fd5b57cc5d2bdcc3704985d951"}]}}
I1027 09:11:04.826794    9945 exec.go:127] defaulting container name to mock
F1027 09:11:04.861602    9945 helpers.go:72] error: Unable to upgrade connection: {
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "tls: handshake did not verify certificate chain",
  "code": 500
}
Comment 2 Jan Chaloupka 2015-10-27 04:32:17 EDT
The same happends with different pod:

# cat ubuntu.yaml
apiVersion: v1
kind: Pod
metadata:
  name: counter
spec:
  containers:
  - name: count
    image: ubuntu:14.04

# kubectl exec counter /bin/sh 
error: Unable to upgrade connection: {
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "tls: handshake did not verify certificate chain",
  "code": 500
}
Comment 3 Jan Chaloupka 2015-10-27 04:50:17 EDT
Based on [1] I have tried to run the scenario with f22 builds (build with golang-1.4.2-3.fc22.x86_64).

# kubectl exec counter /bin/sh -i
error: Error executing remote command: Error executing command in container: container not found ("count")

tls error is gone.

[1] http://stackoverflow.com/questions/32449316/kubernetes-on-mesos-tls-error-on-kubectl-exec
Comment 4 Jan Chaloupka 2015-10-27 05:23:15 EDT
When running the following:

# cat counter.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: counter
spec:
  containers:
  - name: count
    image: ubuntu:14.04
    args: [bash, -c, 
           'for ((i = 0; ; i++)); do echo "$i: $(date)"; sleep 1; done']

# kubectl exec counter /bin/bash -i
uname
Linux
^C^C

Confirming exec command is working is supposed to. 

This is another issue with go-1.5 compiler. I can try to rebuilt f23 with the latest golang-1.5 which is golang-1.5.1-1.fc23. The f23 kubernetes is built with golang-1.5-8.fc23.x86_64
Comment 5 Jan Chaloupka 2015-10-27 12:22:44 EDT
k8s built with golang-1.5.1-1.fc23 suffers the same issue.
Comment 6 Jan Chaloupka 2015-10-27 16:18:51 EDT
Jakub, Vincent,

what is the current state of golang in f23? How are we doing with TLS support?
Comment 7 Jan Chaloupka 2015-10-29 07:49:57 EDT
The issue is fixed in 1.2.0 version of kubernetes.
Comment 9 Fedora Update System 2015-10-29 11:09:00 EDT
kubernetes-1.1.0-0.17.git388061f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-c7fa818f3f
Comment 10 Fedora Update System 2015-11-01 01:57:48 EST
kubernetes-1.1.0-0.17.git388061f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update kubernetes'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-c7fa818f3f
Comment 11 Fedora Update System 2015-11-12 00:22:53 EST
kubernetes-1.1.0-0.17.git388061f.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.