+++ This bug was initially created as a clone of Bug #1273818 +++ Description of problem: Met "x509: cannot validate certificate for x.x.x.x because it doesn't contain any IP SANs" in heapster Pod [chunchen@F17-CCY daily]$ oc get service hawkular-metrics NAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGE hawkular-metrics 172.30.228.45 <none> 443/TCP name=hawkular-metrics 1h [root@openshift-112 ~]# openssl s_client -connect 172.30.228.45:443 CONNECTED(00000003) depth=1 CN = metrics-signer@1446620259 verify error:num=19:self signed certificate in certificate chain verify return:0 --- Certificate chain 0 s:/CN=hawkular-metrics i:/CN=metrics-signer@1446620259 1 s:/CN=metrics-signer@1446620259 i:/CN=metrics-signer@1446620259 --- Server certificate -----BEGIN CERTIFICATE----- MIIDJjCCAhCgAwIBAgIBAjALBgkqhkiG9w0BAQswJDEiMCAGA1UEAwwZbWV0cmlj cy1zaWduZXJAMTQ0NjYyMDI1OTAeFw0xNTExMDQwNjU3NDFaFw0xNzExMDMwNjU3 NDJaMBsxGTAXBgNVBAMTEGhhd2t1bGFyLW1ldHJpY3MwggEiMA0GCSqGSIb3DQEB AQUAA4IBDwAwggEKAoIBAQCtjq9oMZuPufHhJvoUE+4wfJDUwoVXmemRQSc630YY 56f4QWrzAxOWnuiogfoq94q3LnarQ4BGoxHka2cnOHD4TXroWHhYMdI7zrlXL4Su Wat9S4iIKMhQ6IeMiqe2x/BTwLkQA8kBmc5brnbfCF43h1qyhcPz7XsMT+QKAiTM CZlFNzuA8fl7OBX0zkAffz7/wwWW2BglswVR+YHyNARWrMD4CeJjscX6z6LodFOW f2ZAbHpRQCfmgAFwgWvEUJt9wSaWS15ootRIL/nYLbD8XiX5BDX8GMTK81S41zMp VqI9ONLJIoKoGyW2eTTIsebiWPK20AvWEK6yvQFRu+5tAgMBAAGjcDBuMA4GA1Ud DwEB/wQEAwIAoDATBgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMDkG A1UdEQQyMDCCEGhhd2t1bGFyLW1ldHJpY3OCHGhhd2t1bGFyLW1ldHJpY3MuZXhh bXBsZS5jb20wCwYJKoZIhvcNAQELA4IBAQDlRye0pVO4cXYKur40F09IaXpzlyLv g0MOjAJvaWW2HUCJoKtYUMsl9gVQcgnGXSZmtz/gcjM5ZPwlLSvzHrvtn/hfcuu8 P4UmGoHkzvSW181gGF85eegBAvPu2yukw9LbRyn3BTsDRGbnvIk4kQXmg2pjw8DE 24fjlAS838vabwVb3/HqdJxb2f+Wp8+5O+QivjX3EfM0VQsTvcJ/owt4LXdXrPJu L1QbWj6jMf52XMXOFSyJ9gs+D2OmkJft/wCd6t3fQFulsiJ43DT/LHZkNxMkC6Xj uCg5stfGy4yEghLR8EuIy3GLJ4jOtnnn3294Voblw6O0Kb9CEwemKsrF -----END CERTIFICATE----- subject=/CN=hawkular-metrics issuer=/CN=metrics-signer@1446620259 --- No client certificate CA names sent Server Temp Key: DH, 1024 bits --- SSL handshake has read 2366 bytes and written 447 bytes --- New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1 Cipher : DHE-RSA-AES256-SHA Session-ID: 5639C296844FD7874FCBCCE99FD702C0FF9B2F340F2860AF535811035896E699 Session-ID-ctx: Master-Key: 8422EC5971E8EF7FEE6FECBDDB70D2D618D88D62D904032FB2B31CFA6A77CB76137135A7CAD0D6D45249D0D9EC453DD3 Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1446625942 Timeout : 300 (sec) Verify return code: 19 (self signed certificate in certificate chain) --- Version-Release number of selected component (if applicable): oc v3.0.2.905 kubernetes v1.2.0-alpha.1-1107-g4c8e6f4 OSE Puddle: 2015-11-02.1 How reproducible: 100% Steps to Reproduce: 1. Log into openshift server and create a project named "chunpj" 2. Create the Deployer Service Account oc create -f https://raw.githubusercontent.com/openshift/origin-metrics/master/metrics-deployer-setup.yaml 3. Add permissions for service account $ oadm policy add-role-to-user edit system:serviceaccount:chunpj:metrics-deployer $ oadm policy add-cluster-role-to-user cluster-reader system:serviceaccount:chunpj:heapster 4. Create the Hawkular Deployer Secret oc secrets new metrics-deployer nothing=/dev/null 5. Deploy heapster pod via template $ oc process -f https://raw.githubusercontent.com/openshift/origin-metrics/master/metrics.yaml -v HAWKULAR_METRICS_HOSTNAME=hawkular-metrics.example.com,IMAGE_PREFIX=rcm-img-docker01.build.eng.bos.redhat.com:5001/openshift3/,USE_PERSISTENT_STORAGE=false,IMAGE_VERSION=latest,MASTER_URL=https://${MASTER-DNS}:8443 | oc create -f - 6. After deployment is finished, check the heapster pod's log oc logs heapster-lsx6l Actual results: <--------snip----------> I1104 02:13:35.007859 1 kubelet.go:110] about to query kubelet using url: "https://10.14.6.126:10250/stats/metrics/heapster-47vpc/64c50fd8-82c1-11e5-b1a8-fa163e3bdfd8/heapster" I1104 02:13:35.007993 1 kubelet.go:110] about to query kubelet using url: "https://10.14.6.112:10250/stats/xiama/nodejs-example-6-build/285cc753-81ec-11e5-b1a8-fa163e3bdfd8/sti-build" I1104 02:13:35.008201 1 kubelet.go:110] about to query kubelet using url: "https://10.14.6.136:10250/stats/xiuwang/ruby22-sample-build-1-build/58ba5219-8211-11e5-b1a8-fa163e3bdfd8/sti-build" I1104 02:13:35.015008 1 kubelet.go:96] failed to get stats from kubelet url: https://10.14.6.112:10250/stats/xiama/nodejs-example-4-build/5c21a992-81e4-11e5-b1a8-fa163e3bdfd8/sti-build - Get https://10.14.6.112:10250/stats/xiama/nodejs-example-4-build/5c21a992-81e4-11e5-b1a8-fa163e3bdfd8/sti-build: x509: cannot validate certificate for 10.14.6.112 because it doesn't contain any IP SANs I1104 02:13:35.020084 1 kube_pods.go:110] failed to get stats for container "sti-build" in pod "xiama"/"nodejs-example-4-build" I1104 02:13:35.006547 1 kubelet.go:110] about to query kubelet using url: "https://10.14.6.126:10250/stats/dy/ruby22-sample-build-1-build/e0c9cc80-82c2-11e5-b1a8-fa163e3bdfd8/docker-build" I1104 02:13:35.020213 1 kubelet.go:96] failed to get stats from kubelet url: https://10.14.6.127:10250/stats/wsuntest1/ruby-20-centos7-build-1-build/e424b05e-82c0-11e5-b1a8-fa163e3bdfd8/custom-build - Get https://10.14.6.127:10250/stats/wsuntest1/ruby-20-centos7-build-1-build/e424b05e-82c0-11e5-b1a8-fa163e3bdfd8/custom-build: x509: cannot validate certificate for 10.14.6.127 because it doesn't contain any IP SANs I1104 02:13:35.020229 1 kube_pods.go:110] failed to get stats for container "custom-build" in pod "wsuntest1"/"ruby-20-centos7-build-1-build" I1104 02:13:35.020359 1 kubelet.go:96] failed to get stats from kubelet url: https://10.14.6.112:10250/stats/zhouys/database-1-2kskm/d8d07f96-82a0-11e5-b1a8-fa163e3bdfd8/mysql - Get https://10.14.6.112:10250/stats/zhouys/database-1-2kskm/d8d07f96-82a0-11e5-b1a8-fa163e3bdfd8/mysql: x509: cannot validate certificate for 10.14.6.112 because it doesn't contain any IP SANs I1104 02:13:35.020377 1 kube_pods.go:110] failed to get stats for container "mysql" in pod "zhouys"/"database-1-2kskm" I1104 02:13:35.020395 1 kubelet.go:96] failed to get stats from kubelet url: https://10.14.6.127:10250/stats/zhouys/dancer-mysql-example-3-build/d03fd2e7-82a2-11e5-b1a8-fa163e3bdfd8/sti-build - Get https://10.14.6.127:10250/stats/zhouys/dancer-mysql-example-3-build/d03fd2e7-82a2-11e5-b1a8-fa163e3bdfd8/sti-build: x509: cannot validate certificate for 10.14.6.127 because it doesn't contain any IP SANs I1104 02:13:35.020405 1 kube_pods.go:110] failed to get stats for container "sti-build" in pod "zhouys"/"dancer-mysql-example-3-build" I1104 02:13:35.020418 1 kubelet.go:96] failed to get stats from kubelet url: https://10.14.6.136:10250/stats/zhouys/dancer-mysql-example-2-build/a1a6a166-82a2-11e5-b1a8-fa163e3bdfd8/sti-build - Get https://10.14.6.136:10250/stats/zhouys/dancer-mysql-example-2-build/a1a6a166-82a2-11e5-b1a8-fa163e3bdfd8/sti-build: x509: cannot validate certificate for 10.14.6.136 because it doesn't contain any IP SANs I1104 02:13:35.020429 1 kube_pods.go:110] failed to get stats for container "sti-build" in pod "zhouys"/"dancer-mysql-example-2-build" I1104 02:13:35.020441 1 kube_nodes.go:59] Failed to get container stats from Kubelet on node "openshift-149.lab.sjc.redhat.com" I1104 02:13:35.020459 1 kube_nodes.go:59] Failed to get container stats from Kubelet on node "openshift-136.lab.sjc.redhat.com" I1104 02:13:35.020471 1 kube_nodes.go:59] Failed to get container stats from Kubelet on node "openshift-126.lab.sjc.redhat.com" <--------snip----------> Expected results: Should not meet such error in Heapster pod. Additional info: # oc get node NAME LABELS STATUS AGE openshift-112.lab.sjc.redhat.com kubernetes.io/hostname=openshift-112.lab.sjc.redhat.com,perf=y,region=infra,zone=default Ready 1d openshift-126.lab.sjc.redhat.com kubernetes.io/hostname=openshift-126.lab.sjc.redhat.com,region=infra,zone=default Ready 1d openshift-127.lab.sjc.redhat.com kubernetes.io/hostname=openshift-127.lab.sjc.redhat.com,region=infra,zone=default Ready 1d openshift-136.lab.sjc.redhat.com kubernetes.io/hostname=openshift-136.lab.sjc.redhat.com,region=infra,zone=default Ready 1d openshift-149.lab.sjc.redhat.com kubernetes.io/hostname=openshift-149.lab.sjc.redhat.com Ready,SchedulingDisabled 1d
Checked this issue using latest metrics images, it is not reproduced, please refer to below information: [root@openshift-127 ~]# docker images|grep metric | grep rcm rcm-img-docker01.build.eng.bos.redhat.com:5001/openshift3/metrics-deployer latest 981289fe2830 14 hours ago 551.4 MB rcm-img-docker01.build.eng.bos.redhat.com:5001/openshift3/metrics-hawkular-metrics latest b44dc66d64f2 14 hours ago 1.109 GB rcm-img-docker01.build.eng.bos.redhat.com:5001/openshift3/metrics-cassandra latest 8ea21f4b3377 2 weeks ago 472.7 MB rcm-img-docker01.build.eng.bos.redhat.com:5001/openshift3/metrics-heapster latest 800434e62203 2 weeks ago 228.4 MB [chunchen@F17-CCY daily]$ oc logs heapster-tsl34 Starting Heapster with the following arguments: --source=kubernetes:https://openshift-127.lab.sjc.redhat.com:8443?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250 --sink=hawkular:https://hawkular-metrics:443?tenant=_system&labelToTenant=pod_namespace&caCert=/hawkular-cert/hawkular-metrics-ca.certificate&user=hawkular&pass=GkqivlQxQbGIDjX&filter=label(container_name:^/system.slice.*|^/user.slice) --logtostderr=true --tls_cert=/secrets/heapster.cert --tls_key=/secrets/heapster.key --tls_client_ca=/secrets/heapster.client-ca --allowed_users= I1105 03:25:49.269092 1 heapster.go:60] heapster --source=kubernetes:https://openshift-127.lab.sjc.redhat.com:8443?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250 --sink=hawkular:https://hawkular-metrics:443?tenant=_system&labelToTenant=pod_namespace&caCert=/hawkular-cert/hawkular-metrics-ca.certificate&user=hawkular&pass=GkqivlQxQbGIDjX&filter=label(container_name:^/system.slice.*|^/user.slice) --logtostderr=true --tls_cert=/secrets/heapster.cert --tls_key=/secrets/heapster.key --tls_client_ca=/secrets/heapster.client-ca --allowed_users= I1105 03:25:49.269973 1 heapster.go:61] Heapster version 0.18.0 I1105 03:25:49.270735 1 kube_factory.go:168] Using Kubernetes client with master "https://openshift-127.lab.sjc.redhat.com:8443" and version "v1" I1105 03:25:49.270752 1 kube_factory.go:169] Using kubelet port 10250 I1105 03:25:49.271143 1 driver.go:491] Initialised Hawkular Sink with parameters {_system https://hawkular-metrics:443?tenant=_system&labelToTenant=pod_namespace&caCert=/hawkular-cert/hawkular-metrics-ca.certificate&user=hawkular&pass=GkqivlQxQbGIDjX&filter=label(container_name:^/system.slice.*|^/user.slice) 0xc2081985a0 } I1105 03:25:49.734334 1 heapster.go:71] Starting heapster on port 8082 [chunchen@F17-CCY daily]$
I thought https://github.com/openshift/openshift-ansible/pull/609 fixed this for OSE
Yes, that PR adds numbers SANs to master and node certs and would've fixed this issue for any environment installed via ansible.
According to Comment #2 and #4, mark it as verified.