Bug 1709869

Summary:	ssl.CertificateError: hostname 'openshift.default.svc' doesn't match either of 'kubernetes', 'kubernetes.default', 'kubernetes.default.svc'...
Product:	OpenShift Container Platform	Reporter:	Miroslav Novak <mnovak>
Component:	Master	Assignee:	Michal Fojtik <mfojtik>
Status:	CLOSED ERRATA	QA Contact:	Xingxing Xia <xxia>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	4.1.0	CC:	aos-bugs, gblomqui, jokerman, kwills, mchoma, mmccomas, nagrawal, trankin, vlaad, wking
Target Milestone:	---
Target Release:	4.1.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-06-04 10:48:48 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Miroslav Novak 2019-05-14 13:02:01 UTC

Description of problem:
It's no longer possible to make inter pod API calls to openshift.default.svc with /var/run/secrets/kubernetes.io/serviceaccount/ca.crt file. Our python script in pod is getting following error:
CRITICAL:__main__:Cannot query OpenShift API for "https://openshift.default.svc/api/v1/namespaces/mnovak-mynamespace/pods"
Traceback (most recent call last):
  File "/opt/partition/queryosapi.py", line 151, in <module>
    queryResult = getLivingPods()
  File "/opt/partition/queryosapi.py", line 109, in getLivingPods
    jsonPodsData = getPodsJsonData()
  File "/opt/partition/queryosapi.py", line 97, in getPodsJsonData
    jsonText = OpenShiftQuery.queryApi('/api/v1/namespaces/{}/pods'.format(OpenShiftQuery.getNameSpace()))
  File "/opt/partition/queryosapi.py", line 89, in queryApi
    return urllib2.urlopen(request, cafile = OpenShiftQuery.CERT_FILE_PATH).read()
  File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 1258, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib64/python2.7/urllib2.py", line 1211, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1041, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1075, in _send_request
    self.endheaders(body)
  File "/usr/lib64/python2.7/httplib.py", line 1037, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 881, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.7/httplib.py", line 843, in send
    self.connect()
  File "/usr/lib64/python2.7/httplib.py", line 1263, in connect
    ssl.match_hostname(self.sock.getpeercert(), server_hostname)
  File "/usr/lib64/python2.7/ssl.py", line 267, in match_hostname
    % (hostname, ', '.join(map(repr, dnsnames))))
ssl.CertificateError: hostname 'openshift.default.svc' doesn't match either of 'kubernetes', 'kubernetes.default', 'kubernetes.default.svc', 'kubernetes.default.svc.cluster.local', '172.30.0.1'

This seems to be change against OCP 3.11 where it works.

Version-Release number of selected component (if applicable):
4.1.0-rc.3

How reproducible - always:
Run from inside a pod:
curl  --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer dBNsUYunIES1nIB0yfL-NNSE51Nk_cRXcDSzKz2dEuo" 'https://openshift.default.svc/api/v1/namespaces/mnovak-mynamespace/pods'

Comment 1 mchoma 2019-05-14 13:20:19 UTC

I assume problem will be that certificate for kube api server does not contains `openshift.default.svc` as Subject Alternative name. 

curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt https://openshift.default.svc
curl: (51) Unable to communicate securely with peer: requested domain name does not match the server's certificate.

Comment 4 Michal Fojtik 2019-05-14 15:10:15 UTC

Fix: https://github.com/openshift/installer/pull/1750

Comment 5 Michal Fojtik 2019-05-14 15:29:45 UTC

Additionally: https://github.com/openshift/cluster-kube-apiserver-operator/pull/473

Comment 6 W. Trevor King 2019-05-14 20:28:35 UTC

> It's no longer possible to make inter pod API calls to openshift.default.svc with /var/run/secrets/kubernetes.io/serviceaccount/ca.crt file. Our python script in pod is getting following error:

What's the impact of this?  It's not my call, but I'm trying to understand if this should block 4.1.0, or if it's ok to land in 4.1.z, or if we can skip it for 4.1 entirely.

Comment 7 Ken Wills 2019-05-14 22:39:00 UTC

openshift.default.svc has been in the OpenShift docs in a couple of places for a long time, for example https://docs.openshift.com/container-platform/3.11/dev_guide/service_accounts.html#using-a-service-accounts-credentials-inside-a-container and https://docs.openshift.com/container-platform/3.11/architecture/additional_concepts/authentication.html#oauth-server-metadata 

So I think as it was documented behavior, it should block 4.1.0.

Comment 10 W. Trevor King 2019-05-15 01:02:27 UTC

This bug targets 4.1.0 (comment 7 makes sense to me, but again, not my call), and [1] is still open.  Back to POST.

[1]: https://github.com/openshift/installer/pull/1751

Comment 11 Michal Fojtik 2019-05-16 08:30:28 UTC

4.1 is merged, moving to QA.

Comment 14 Xingxing Xia 2019-05-20 01:42:13 UTC

Verified in latest payload 4.1.0-0.nightly-2019-05-18-050636 env:
$ oc rsh mysql-1-2dlfh
sh-4.2$ curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer cuHZWeyX..." 'https://openshift.d
efault.svc/api/v1/namespaces/testuser-37-proj/pods'
{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
...

Comment 16 errata-xmlrpc 2019-06-04 10:48:48 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758