Bug 1708640 - Not able to query jolokia on jboss images
Summary: Not able to query jolokia on jboss images
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Auth
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Stefan Schimanski
QA Contact: Chuan Yu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-05-10 12:48 UTC by mchoma
Modified: 2019-08-26 10:20 UTC (History)
23 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-08-26 10:20:57 UTC


Attachments (Terms of Use)
pod.log (62.07 KB, application/octet-stream)
2019-05-14 08:17 UTC, mchoma
no flags Details


Links
System ID Priority Status Summary Last Updated
JBoss Issue Tracker JBEAP-16845 Critical Reopened [OCP 4.1] Not able to query jolokia on jboss images 2019-10-21 15:07:43 UTC

Description mchoma 2019-05-10 12:48:04 UTC
Description of problem:
We are not able to query jolokia on OCP 4.1 as we used to be on OCP 3.11. Here is what we did

Version-Release number of selected component (if applicable):
4.1

How reproducible:
always

Steps to Reproduce:
{code}
# Start EAP 
oc import-image jboss-eap-7-tech-preview/eap-cd-openshift --from=registry.access.redhat.com/jboss-eap-7-tech-preview/eap-cd-openshift --confirm
oc new-app eap-cd-openshift --name jolokia-reproducer
oc expose svc/jolokia-reproducer

# Query jolokia
# Command contains pod name which has to be changed and token which you can get with `oc whoami -t`
curl -v -k --oauth2-bearer GNOw3jTbgWu143pF21b8cSSG5Mksr_0t8_ZLG4zRrXI https://api.eap-qe-ocp41-cluster.eap-qe-ocp41-cluster.fw.rhcloud.com:6443/api/v1/namespaces/mchoma/pods/https:jolokia-reproducer-1-xnjrs:8778/proxy/jolokia/

Actual results:
*   Trying 3.14.209.123...
* TCP_NODELAY set
* Connected to api.eap-qe-ocp41-cluster.eap-qe-ocp41-cluster.fw.rhcloud.com (3.14.209.123) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=api.eap-qe-ocp41-cluster.eap-qe-ocp41-cluster.fw.rhcloud.com
*  start date: May  2 06:49:44 2019 GMT
*  expire date: Jun  1 06:49:45 2019 GMT
*  issuer: OU=openshift; CN=kube-apiserver-lb-signer
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Server auth using Bearer with user ''
* Using Stream ID: 1 (easy handle 0x562191462590)
> GET /api/v1/namespaces/mchoma/pods/https:jolokia-reproducer-1-xnjrs:8778/proxy/jolokia/ HTTP/2
> Host: api.eap-qe-ocp41-cluster.eap-qe-ocp41-cluster.fw.rhcloud.com:6443
> Authorization: Bearer GNOw3jTbgWu143pF21b8cSSG5Mksr_0t8_ZLG4zRrXI
> User-Agent: curl/7.61.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 2000)!
< HTTP/2 401 
< audit-id: e8a1815b-9901-4a87-910d-476e38ea8f5c
< cache-control: no-store
< date: Thu, 02 May 2019 11:34:23 GMT
< www-authenticate: Basic realm="jolokia"
< content-length: 0
< 
* Connection #0 to host api.eap-qe-ocp41-cluster.eap-qe-ocp41-cluster.fw.rhcloud.com left intact
{code}

Problem is request is not authenticated.

Expected results:
Similar command in OCP 3.11 authenticates and returns something

{code}
curl -v -k --oauth2-bearer 10n_P-FK6ssm0RjY_1oJKDc3Yr9csThrLRJ9XP1viwA https://api.all-in-one-034.dynamic.xpaas:8443/api/v1/namespaces/mchoma/pods/https:eap-cd-openshift-1-j9dd5:8778/proxy/jolokia/
*   Trying 10.0.76.171...
* TCP_NODELAY set
* Connected to api.all-in-one-034.dynamic.xpaas (10.0.76.171) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=10.0.76.171
*  start date: Apr 12 08:05:02 2019 GMT
*  expire date: Apr 11 08:05:03 2021 GMT
*  issuer: CN=openshift-signer@1555056302
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Server auth using Bearer with user ''
* Using Stream ID: 1 (easy handle 0x561c92508590)
> GET /api/v1/namespaces/mchoma/pods/https:eap-cd-openshift-1-j9dd5:8778/proxy/jolokia/ HTTP/2
> Host: api.all-in-one-034.dynamic.xpaas:8443
> Authorization: Bearer 10n_P-FK6ssm0RjY_1oJKDc3Yr9csThrLRJ9XP1viwA
> User-Agent: curl/7.61.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200 
< cache-control: no-store
< cache-control: no-cache
< content-type: text/plain; charset=utf-8
< date: Thu, 02 May 2019 11:26:47 GMT
< expires: Thu, 02 May 2019 10:26:47 GMT
< pragma: no-cache
< content-length: 848
< 
* Connection #0 to host api.all-in-one-034.dynamic.xpaas left intact
{"request":{"type":"version"},"value":{"agent":"1.5.0","protocol":"7.2","config":{"listenForHttpService":"true","maxCollectionSize":"0","authIgnoreCerts":"false","agentId":"10.128.1.52-490-6e0be858-jvm","agentType":"jvm","policyLocation":"classpath:\/jolokia-access.xml","agentContext":"\/jolokia","mimeType":"text\/plain","discoveryEnabled":"false","streaming":"true","password":"VXey5eTaxdKwhNozLqnUtSf5vCNcqw","historyMaxEntries":"10","allowDnsReverseLookup":"true","maxObjects":"0","debug":"false","serializeException":"false","maxDepth":"15","authMode":"basic","canonicalNaming":"true","allowErrorDetails":"true","realm":"jolokia","includeStackTrace":"true","user":"jolokia","useRestrictorService":"false","debugMaxEntries":"100"},"info":{"product":"JBoss EAP CD","vendor":"RedHat","version":"7.3.0.CD15"}},"timestamp":1556796407,"status":200}
{code}

Additional info:
I have seen this also on amq image in tests.

Comment 1 Frederic Branczyk 2019-05-10 13:15:03 UTC
I don't think this belongs to the OCP Monitoring component. We take care of the cluster monitoring stack and Prometheus itself. Component and product teams are responsible for their monitoring themselves. Who put that component there, and who's responsible for jolokia?

Comment 2 mchoma 2019-05-10 13:36:20 UTC
In that case I agree Monitoring was bad choice. I was thinking about several components: Auth, Networking, Pod, Security ...

Could someone help me to redirect issue to proper component then?

Which component is involved in accessing through api https://api.$CLUSTER:6443/api/v1/namespaces/namespace/pods/https:$POD:8778/proxy/ ?

Comment 3 Frederic Branczyk 2019-05-10 13:39:53 UTC
I'd say auth. Although, it looks like this may just be a setup/permission issue, you may just not have enough permissions to use the proxy subresource of that pod. I'd check the audit log and/or apiserver log to see whether this was an RBAC failure.

Comment 4 mchoma 2019-05-10 13:45:03 UTC
I can check that on Monday. Although I was using as kubeadmin, which is most powerful user on our installation, so I doubt that could be the case.

Comment 6 mchoma 2019-05-10 14:03:13 UTC
It is possible our test in time of creation was impacted by these resources [1] [2] [3], which describes that way of usage. If this is no more supported way we can look into alternative ways. Could you elaborate more? Are you talking about exposing pod:8787 through service to get public access point?

[1] https://access.redhat.com/documentation/en-us/red_hat_jboss_a-mq/6.3/html/red_hat_jboss_a-mq_for_openshift/tutorials#monitoring_a_mq
[2] https://developers.redhat.com/blog/2016/03/30/jolokia-jvm-monitoring-in-openshift/#error=interaction_required&state=6d2c29b9-d7d7-4229-887e-6e43d26f9655
[3] https://access.redhat.com/solutions/3756021

Comment 7 Clayton Coleman 2019-05-10 15:34:44 UTC
Hold on.  The proxy is part of our public API.  It must work.  If we broke it in 4.1 it must be fixed.

Proxy is a V1, public, supported API.

Comment 9 David Eads 2019-05-10 17:22:46 UTC
I deleted the token, so don't freak out.  I tried this an it worked fine.

[deads@deads-02 installer]$ curl -v  -k --oauth2-bearer N-tLptkeh71tpwHffvxRMStClh_Ym5d807YHYDPlC_g https://api.ci-ln-1nvz24k-d5d6b.origin-ci-int-aws.dev.rhcloud.com:6443/api/v1/namespaces/default/pods/https:jolokia-reproducer-1-v8xh2:8778/proxy/jolokia
*   Trying 3.213.152.204...
* TCP_NODELAY set
* Connected to api.ci-ln-1nvz24k-d5d6b.origin-ci-int-aws.dev.rhcloud.com (3.213.152.204) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=api.ci-ln-1nvz24k-d5d6b.origin-ci-int-aws.dev.rhcloud.com
*  start date: May 10 16:54:12 2019 GMT
*  expire date: Jun  9 16:54:13 2019 GMT
*  issuer: OU=openshift; CN=kube-apiserver-lb-signer
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Server auth using Bearer with user ''
* Using Stream ID: 1 (easy handle 0x562326501630)
> GET /api/v1/namespaces/default/pods/https:jolokia-reproducer-1-v8xh2:8778/proxy/jolokia HTTP/2
> Host: api.ci-ln-1nvz24k-d5d6b.origin-ci-int-aws.dev.rhcloud.com:6443
> Authorization: Bearer N-tLptkeh71tpwHffvxRMStClh_Ym5d807YHYDPlC_g
> User-Agent: curl/7.61.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 2000)!
< HTTP/2 404 
< audit-id: 74397f07-c82c-4f95-a3ef-9a2eb778b4f8
< cache-control: no-store
< content-type: text/html
< content-length: 50
< date: Fri, 10 May 2019 17:12:57 GMT
< 
* Connection #0 to host api.ci-ln-1nvz24k-d5d6b.origin-ci-int-aws.dev.rhcloud.com left intact
<h1>404 Not Found</h1>No context found for request[deads@deads-02 installer]$ 




Now as for the specific issue you saw.  The 401 you saw in your curl was actually a 401 from your application, not from the kube-apiserver. You were authenticated to the kube-apiserver and because this endpoint isn't a real proxy (as Ben pointed out about), you cannot pass a secondary token to it.  Your app (jolokia I guess), didn't get any authentication information and replied with a 401.  It stands out because `www-authenticate: Basic realm="jolokia"` is a challenge header and our kube-apiserver doesn't accept basic auth and so it doesn't reply with challenge headers.  And it doesn't know anything about a jolokia realm.


Notice the difference if you use a bad token:

curl -v  -k --oauth2-bearer bad-token https://api.ci-ln-1nvz24k-d5d6b.origin-ci-int-aws.dev.rhcloud.com:6443/api/v1/namespaces/default/pods/https:jolokia-reproducer-1-v8xh2:8778/proxy/jolokia
*   Trying 3.213.152.204...
* TCP_NODELAY set
* Connected to api.ci-ln-1nvz24k-d5d6b.origin-ci-int-aws.dev.rhcloud.com (3.213.152.204) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=api.ci-ln-1nvz24k-d5d6b.origin-ci-int-aws.dev.rhcloud.com
*  start date: May 10 16:54:12 2019 GMT
*  expire date: Jun  9 16:54:13 2019 GMT
*  issuer: OU=openshift; CN=kube-apiserver-lb-signer
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Server auth using Bearer with user ''
* Using Stream ID: 1 (easy handle 0x55ab3c901630)
> GET /api/v1/namespaces/default/pods/https:jolokia-reproducer-1-v8xh2:8778/proxy/jolokia HTTP/2
> Host: api.ci-ln-1nvz24k-d5d6b.origin-ci-int-aws.dev.rhcloud.com:6443
> Authorization: Bearer bad-token
> User-Agent: curl/7.61.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 2000)!
< HTTP/2 401 
< audit-id: 8275b14e-076d-44aa-a768-a28ccb3e9fc9
< cache-control: no-store
< content-type: application/json
< content-length: 165
< date: Fri, 10 May 2019 17:21:06 GMT
< 
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
* Connection #0 to host api.ci-ln-1nvz24k-d5d6b.origin-ci-int-aws.dev.rhcloud.com left intact
}[deads@deads-02 installer]$ 


There is no challenge header and our 401 messages have a json format in the body.



If you want us to have another look, please use an different app, something simple that doesn't require secondary authentication.

Comment 10 mchoma 2019-05-13 06:04:34 UTC
Thanks for clarification.

But again, do you understand why this scenario works on OCP 3.11 and stopped to work on OCP 4.1? Which behaviour is correct?

Comment 11 mchoma 2019-05-14 07:40:42 UTC
We are discussing this in paralel on https://issues.jboss.org/browse/JBEAP-16845.

It seems to us proxy -> jolokia communication is secured by CLIENT_CERT authentication. As jolokia is secured with self signed certificate theory is OCP 4.1 is not trusting such certificate. Was proxy on OCP 3.11 skipping cert validation? Or is still default client certifikace for kubeapiserver cn=system:master-proxy?
But we are missing any log information to confirm our deductions. Could you advice how we can see additional logs from proxy ideally TLS handshake log from proxy similar to -Djavax.net.debug=all?

Authentication in jolokia is configured like this:
sh-4.2$ cat /opt/jolokia/etc/jolokia.properties
host=*
port=8778
discoveryEnabled=false
user=jolokia
password=gTa34RrujeddqzxcT3ZnRCmoTrMfe3
protocol=https
useSslClientAuthentication=true
extraClientCheck=true
caCert=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
clientPrincipal=cn=system:master-proxy
sh-4.2$

Comment 12 mchoma 2019-05-14 08:15:58 UTC
I have managed to get TLS handshake detail log (pod.log) on eap side by setting environment variable `JAVA_OPTS_APPEND=-Djavax.net.debug=all`.

From that log I see kube api server does not provide client certificate
{code}
[0m[0m07:57:39,050 INFO  [stdout] (Thread-97) *** Certificate chain
[0m[0m07:57:39,050 INFO  [stdout] (Thread-97) <Empty>
[0m[0m07:57:39,050 INFO  [stdout] (Thread-97) ***
{code}

Which could be caused by fact kube api server client certificate does not comply to what jolokia is requesting
{code}
[0m[0m07:57:38,860 INFO  [stdout] (Thread-97) Cert Authorities:
[0m[0m07:57:38,860 INFO  [stdout] (Thread-97) <CN=Jolokia Agent 1.5.0, OU=JVM, O=jolokia.org, L=Pegnitz, ST=Franconia, C=DE>
[0m[0m07:57:38,860 INFO  [stdout] (Thread-97) <CN=kube-apiserver-lb-signer, OU=openshift>
{code}

I see kube-apiserver-lb-signer is from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt which is used by jolokia as ca cert. Question now is what is kube api server using as default client certificate on OCP 4.1? Apparently that is not signed by kube-apiserver-lb-signer. So that is why it stopped work on OCP 4.1.

Comment 13 mchoma 2019-05-14 08:17:41 UTC
Created attachment 1568314 [details]
pod.log

Comment 16 mchoma 2019-05-15 06:09:14 UTC
Reopening as it seems to me this needs more attention. Seems to me certificate which proxy(kube api server) use for CLIENT_CERT authentication does not match with truststore /var/run/secrets/kubernetes.io/serviceaccount/ca.crt.  

Where can I find proxy(kube api server) certificate?

Comment 27 Antonin Stefanutti 2019-05-24 16:11:28 UTC
I don't know if that changed in OpenShift 4.x, but in 3.x, the client certificate used by Jolokia is the one configured in master-config.yaml, that allows setting a client cert for the proxy to present when proxying using TLS:

kubernetesMasterConfig:
    ...
    certFile: master.proxy-client.crt
    keyFile: master.proxy-client.key

FWIW, I've been able to reproduce the issue on OpenShift 3.11.0.

While the same test works with OpenShift 3.10.0:

$ minishift start --openshift-version=v3.10.0
$ kubectl get --raw /api/v1/namespaces/fuse/pods/https:s2i-fuse73-spring-boot-camel-2-rdqq8:8778/proxy/jolokia/ --v=9 
I0524 16:44:14.764310   91448 loader.go:359] Config loaded from file /Users/astefanu/.kube/config
I0524 16:44:14.799425   91448 round_trippers.go:419] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.14.2 (darwin/amd64) kubernetes/66049e3" -H "Authorization: Bearer cC7M7_vUiQrRxmAF-GhPwPa7d0txaV1QmIOhkTK5KMQ" 'https://192.168.64.71:8443/api/v1/namespaces/fuse/pods/https:s2i-fuse73-spring-boot-camel-2-rdqq8:8778/proxy/jolokia/'
I0524 16:44:15.394543   91448 round_trippers.go:438] GET https://192.168.64.71:8443/api/v1/namespaces/fuse/pods/https:s2i-fuse73-spring-boot-camel-2-rdqq8:8778/proxy/jolokia/ 200 OK in 595 milliseconds
I0524 16:44:15.394577   91448 round_trippers.go:444] Response Headers:
I0524 16:44:15.394587   91448 round_trippers.go:447]     Cache-Control: no-store
I0524 16:44:15.394635   91448 round_trippers.go:447]     Cache-Control: no-cache
I0524 16:44:15.394655   91448 round_trippers.go:447]     Content-Type: text/plain; charset=utf-8
I0524 16:44:15.394662   91448 round_trippers.go:447]     Date: Fri, 24 May 2019 14:44:30 GMT
I0524 16:44:15.394668   91448 round_trippers.go:447]     Expires: Fri, 24 May 2019 13:44:30 GMT
I0524 16:44:15.394673   91448 round_trippers.go:447]     Pragma: no-cache
I0524 16:44:15.394678   91448 round_trippers.go:447]     Content-Length: 780
{"request":{"type":"version"},"value":{"agent":"1.5.0","protocol":"7.2","config":{"listenForHttpService":"true","maxCollectionSize":"0","authIgnoreCerts":"false","agentId":"172.17.0.8-1-6d6f6e28-jvm","agentType":"jvm","policyLocation":"classpath:\/jolokia-access.xml","agentContext":"\/jolokia","mimeType":"text\/plain","discoveryEnabled":"false","streaming":"true","password":"CxUUoVagLOAIGUTLi5FnYKn0gPNr0c","historyMaxEntries":"10","allowDnsReverseLookup":"true","maxObjects":"0","debug":"false","serializeException":"false","maxDepth":"15","authMode":"basic","canonicalNaming":"true","allowErrorDetails":"true","realm":"jolokia","includeStackTrace":"true","user":"jolokia","useRestrictorService":"false","debugMaxEntries":"100"},"info":{}},"timestamp":1558709070,"status":200}% 

With 3.11.0, the client certificate presented to Jolokia by the API proxy is empty as reported with JAVA_OPTIONS=-Djavax.net.debug=all: 

*** Certificate chain
<Empty>
***

With 3.10.0, the client certificate is presented as expected:

*** Certificate chain
chain [0] = [
[
  Version: V3
  Subject: CN=system:master-proxy
  Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11
  Key:  Sun RSA public key, 2048 bits
  modulus: 24538824463296429476158344027594341222983297637389985686887239626224123943539183061884281786478584619787095828890492968667750261339004288466315046775944846036259902977051175812769295285797591920694908406110083775657031670360073456304933676685464354372471723029633162734612235584266478091096909553121514228176976984963272813650863854915581269997667245569268972451538541446020787147645568489329212986241076399956143008390668266104878254701988032248727491492858261198111842986990701298510005873073307331156245102739091290424276784359738653894539831173743994305380255566487472854693996012479501593989350720180343982383703
  public exponent: 65537
  Validity: [From: Fri May 24 08:45:58 UTC 2019,
               To: Sun May 23 08:45:59 UTC 2021]
  Issuer: CN=openshift-signer@1558687558
  SerialNumber: [    05]
Certificate Extensions: 3
[1]: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:false
  PathLen: undefined
]
[2]: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  clientAuth
]
[3]: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
  Key_Encipherment
]
]

It is clear, that with an empty certificate being presented, Jolokia SSL client authentication will fail and may fallback to username / password challenge if enabled.

Comment 30 Xingxing Xia 2019-05-29 09:21:43 UTC
The notes LGTM

Comment 31 Xingxing Xia 2019-05-30 07:57:42 UTC
It is OK to make above document notes. But the issue should be fixed. Changing bug fields.


Note You need to log in before you can comment on or make changes to this bug.