Reassigning as this is not a logging problem.  The customer does not appear to want to use the recommended method of using separate certificates for external and internal services.

The kibana-proxy talks to the master at https://kubernetes.default.svc.cluster.local which is a cluster-internal address resolved by SkyDNS to an internal IP. It uses the /var/run/secrets/kubernetes.io/serviceaccount/ca.crt supplied to every pod by kubernetes for the purpose of talking to the master at its internal address.

I'm actually still a little confused about why the solution described here doesn't work for the customer:
https://docs.openshift.com/container-platform/3.5/install_config/certificate_customization.html#configuring-custom-certificates

Even if they're wanting to use a single address for the external master API and for the nodes and other infrastructure to talk to, because kibana-proxy is making the request against kubernetes.default.svc.cluster.local the master should still be able to serve that address separately with the OpenShift-generated internal cert validated by the internal CA. So if their master config ends up like the following I don't see the problem:

servingInfo:
  bindAddress: 0.0.0.0:8443
  bindNetwork: tcp4
  certFile: master.server.crt
  clientCA: ca-bundle.crt
  keyFile: master.server.key
  maxRequestsInFlight: 500
  requestTimeoutSeconds: 3600
  namedCertificates:
  - certFile: custom.crt
    keyFile: custom.key
    names:
    - "master.customer.com"

I'm also surprised there isn't similarly a problem with the router which I believe also watches the master at this internal address to build its routing tables (I could be off though).

But, let's suppose we wanted to reconfigure kibana-proxy to make the master request using the same address as everything else. This is possible with some reconfiguration to have kibana-proxy use a different address and CA.

The address is actually the easiest thing to change; there's a parameter OAP_MASTER_URL in the deployer kibana-proxy template (see https://github.com/openshift/origin-aggregated-logging/blob/release-1.4/deployer/templates/kibana.yaml#L112) which can follows the deployer parameters MASTER_URL (see https://github.com/openshift/origin-aggregated-logging/blob/release-1.4/deployer/deployer.yaml#L154).

The CA is a bit more tricky. I suspect that updating the master-config.yaml to replace ca-bundle.crt with their custom CA chain should work, but I've never tried it (and don't know if there's an ansible variable to achieve this). But that would probably be the most thorough solution if it worked, as other pods would have access to the correct CA not just kibana-proxy.

To update the kibana-proxy specifically, you would need to add a secret (or an entry to an existing secret) to provide the CA bundle. Then you would need to update the environment variable OAP_MASTER_CA_FILE on the kibana-proxy container template (you can see where it is defined now at https://github.com/openshift/origin-aggregated-logging/blob/release-1.4/deployer/templates/kibana.yaml#L121) to point where the secret has mounted it. There is definitely no deployer/ansible parameter for doing this, it would be an after-deployment modification.

Having done all that, it occurs to me that Elasticsearch will need the exact same treatment because it too queries the master with the auth token that it gets. It is not at all apparent to me how or if configuration there is even possible; it's using kubernetes-client which seems to have the values hardcoded. https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/Config.java#L110

I have tested with 6.11.0 [1] and I don't have this problem. Also tested with tag 0.1.0 using nodejs 4.7.2 [2] but it didn't work.

[1] https://github.com/ruromero/openshift-auth-proxy/tree/update-pkg
[2] https://github.com/fabric8io/openshift-auth-proxy/tree/0.1.0

What inputs were provided to the ansible configuration?

1. Did they override the signing CA?
2. Did they provide a custom CA bundle?
3. Were certificates for internal hostnames regenerated as well?

(In reply to Jordan Liggitt from comment #7)
> What inputs were provided to the ansible configuration?
> 
> 1. Did they override the signing CA?
> 2. Did they provide a custom CA bundle?
> 3. Were certificates for internal hostnames regenerated as well?

They redeployed certificates using a custom CA.

https://docs.openshift.com/container-platform/3.5/install_config/redeploying_certificates.html#redeploying-new-custom-ca

I honestly don't know the exact reason but it has been confirmed to be solved after upgrading to nodeJS 4.8.1 

https://github.com/fabric8io/openshift-auth-proxy/pull/21

Not auth related, moving to Kibana (Logging)

Setting the following env variable on kibana-proxy is enough to avoid the logging loop

NODE_TLS_REJECT_UNAUTHORIZED: 0

If I have read the documentation correctly, this value will ignore self-signed certificates.

What I don't understand is why with the default value for OAP_MASTER_CA_FILE which is the /var/run/secrets/kubernetes.io/serviceaccount/ca.crt is not working
  https://github.com/fabric8io/openshift-auth-proxy/blob/master/lib/config.js#L77

Thank you for your comments beforehand.

> NODE_TLS_REJECT_UNAUTHORIZED: 0

> If I have read the documentation correctly, this value will ignore self-signed > certificates.

No! This value will ignore certificate validity entirely, accepting anything.
You should *never* use this option except for some debugging/development reason.

We clearly need a CI job that deploys an uses an "external" CA, so we can validate these code paths.

@Eric,

Reassigning to you to add any input to #c22.  Please comment as appropriate and reclose if it makes sense.

Reclosing given no activity from the customer who was redeploying certs 2 months prior.

I think this issue can be chalked up to something related to https://access.redhat.com/solutions/3805841. Customer should probably use openshift_additiona_ca to configure trust between oauth-proxy and k8s api.

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days