Bug 1709869 - ssl.CertificateError: hostname 'openshift.default.svc' doesn't match either of 'kubernetes', 'kubernetes.default', 'kubernetes.default.svc'...
Summary: ssl.CertificateError: hostname 'openshift.default.svc' doesn't match either o...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Master
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.1.0
Assignee: Michal Fojtik
QA Contact: Xingxing Xia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-05-14 13:02 UTC by Miroslav Novak
Modified: 2019-06-04 10:49 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-04 10:48:48 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 None None None 2019-06-04 10:49:49 UTC
Github openshift cluster-kube-apiserver-operator pull 479 None None None 2019-05-17 22:19:00 UTC

Description Miroslav Novak 2019-05-14 13:02:01 UTC
Description of problem:
It's no longer possible to make inter pod API calls to openshift.default.svc with /var/run/secrets/kubernetes.io/serviceaccount/ca.crt file. Our python script in pod is getting following error:
CRITICAL:__main__:Cannot query OpenShift API for "https://openshift.default.svc/api/v1/namespaces/mnovak-mynamespace/pods"
Traceback (most recent call last):
  File "/opt/partition/queryosapi.py", line 151, in <module>
    queryResult = getLivingPods()
  File "/opt/partition/queryosapi.py", line 109, in getLivingPods
    jsonPodsData = getPodsJsonData()
  File "/opt/partition/queryosapi.py", line 97, in getPodsJsonData
    jsonText = OpenShiftQuery.queryApi('/api/v1/namespaces/{}/pods'.format(OpenShiftQuery.getNameSpace()))
  File "/opt/partition/queryosapi.py", line 89, in queryApi
    return urllib2.urlopen(request, cafile = OpenShiftQuery.CERT_FILE_PATH).read()
  File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 1258, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib64/python2.7/urllib2.py", line 1211, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1041, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1075, in _send_request
    self.endheaders(body)
  File "/usr/lib64/python2.7/httplib.py", line 1037, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 881, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.7/httplib.py", line 843, in send
    self.connect()
  File "/usr/lib64/python2.7/httplib.py", line 1263, in connect
    ssl.match_hostname(self.sock.getpeercert(), server_hostname)
  File "/usr/lib64/python2.7/ssl.py", line 267, in match_hostname
    % (hostname, ', '.join(map(repr, dnsnames))))
ssl.CertificateError: hostname 'openshift.default.svc' doesn't match either of 'kubernetes', 'kubernetes.default', 'kubernetes.default.svc', 'kubernetes.default.svc.cluster.local', '172.30.0.1'

This seems to be change against OCP 3.11 where it works.

Version-Release number of selected component (if applicable):
4.1.0-rc.3

How reproducible - always:
Run from inside a pod:
curl  --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer dBNsUYunIES1nIB0yfL-NNSE51Nk_cRXcDSzKz2dEuo" 'https://openshift.default.svc/api/v1/namespaces/mnovak-mynamespace/pods'

Comment 1 mchoma 2019-05-14 13:20:19 UTC
I assume problem will be that certificate for kube api server does not contains `openshift.default.svc` as Subject Alternative name. 

curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt https://openshift.default.svc
curl: (51) Unable to communicate securely with peer: requested domain name does not match the server's certificate.

Comment 4 Michal Fojtik 2019-05-14 15:10:15 UTC
Fix: https://github.com/openshift/installer/pull/1750

Comment 5 Michal Fojtik 2019-05-14 15:29:45 UTC
Additionally: https://github.com/openshift/cluster-kube-apiserver-operator/pull/473

Comment 6 W. Trevor King 2019-05-14 20:28:35 UTC
> It's no longer possible to make inter pod API calls to openshift.default.svc with /var/run/secrets/kubernetes.io/serviceaccount/ca.crt file. Our python script in pod is getting following error:

What's the impact of this?  It's not my call, but I'm trying to understand if this should block 4.1.0, or if it's ok to land in 4.1.z, or if we can skip it for 4.1 entirely.

Comment 7 Ken Wills 2019-05-14 22:39:00 UTC
openshift.default.svc has been in the OpenShift docs in a couple of places for a long time, for example https://docs.openshift.com/container-platform/3.11/dev_guide/service_accounts.html#using-a-service-accounts-credentials-inside-a-container and https://docs.openshift.com/container-platform/3.11/architecture/additional_concepts/authentication.html#oauth-server-metadata 

So I think as it was documented behavior, it should block 4.1.0.

Comment 10 W. Trevor King 2019-05-15 01:02:27 UTC
This bug targets 4.1.0 (comment 7 makes sense to me, but again, not my call), and [1] is still open.  Back to POST.

[1]: https://github.com/openshift/installer/pull/1751

Comment 11 Michal Fojtik 2019-05-16 08:30:28 UTC
4.1 is merged, moving to QA.

Comment 14 Xingxing Xia 2019-05-20 01:42:13 UTC
Verified in latest payload 4.1.0-0.nightly-2019-05-18-050636 env:
$ oc rsh mysql-1-2dlfh
sh-4.2$ curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer cuHZWeyX..." 'https://openshift.d
efault.svc/api/v1/namespaces/testuser-37-proj/pods'
{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
...

Comment 16 errata-xmlrpc 2019-06-04 10:48:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758


Note You need to log in before you can comment on or make changes to this bug.