Description of problem: Unless you configure an RSYSLOG logging mechanism, http logs are not saved on the routers, or in a location where http requests can be seen or used for debugging. If you look at the router configuration you can see that logging for http routes is only configured, if RSYSLOG logging mechanism are configured. https://github.com/openshift/origin/blob/master/images/router/haproxy/conf/haproxy-config.template#L12-L14 https://github.com/openshift/origin/blob/master/images/router/haproxy/conf/haproxy-config.template#L39-L43 Expected results: http routing (access logs) should need be logged with http request. Ideally these logs should be send to the default logging stack - or be be capable of logging to the logging stack (by project if possible). Additional info: http://unix.stackexchange.com/questions/260940/make-haproxy-log-to-a-local-file-instead-of-syslog
The process for manually enabling this with a syslog endpoint is https://docs.openshift.com/container-platform/3.4/admin_guide/router.html#viewing-logs
We should be able to use fluentd as a syslog endpoint for the router, using https://docs.fluentd.org/v0.12/articles/in_syslog
*** Bug 1509525 has been marked as a duplicate of this bug. ***
Discussing with mbarrett, he believes it has been covered with previous discussions with eparis. Setting to networking team to confirm.
Does ROUTER_SYSLOG_ADDRESS and ROUTER_SYSLOG_FORMAT satisfy their need: https://github.com/openshift/openshift-docs/blob/master/admin_guide/router.adoc#viewing-logs
(In reply to Ben Bennett from comment #15) > Does ROUTER_SYSLOG_ADDRESS and ROUTER_SYSLOG_FORMAT satisfy their need: > > https://github.com/openshift/openshift-docs/blob/master/admin_guide/router. > adoc#viewing-logs No, because it's not aggregated per application, in the logging solution we deploy! In short, the goal is that a customer can get access logs for an application, based on the application/project/route they have developed. Think about trying to debug/look at issues with online/dedicated?
Getting router access logs in front of users in aggregated logging requires two high-level steps: 1. Generating the logs with necessary metadata to determine which project they're for. 2. Putting the logs where they can be ingested by the EFK stack. First let's distinguish between "router logs" and "access logs". The logs the router outputs at startup, when refreshing endpoints, and so forth are the "router logs" currently produced and probably not of much interest to the application owner; more of an ops concern. At the moment AFAIK we don't generate "access logs" from haproxy -- these would be logs like the ones we're familiar with from apache recording each request that comes in and what happened with it. These would be interesting to the app owner who wants to track why a request doesn't make it to their pod, or why it got mangled before/after. If I understand the RFE, that's what we're interested in here. And it's worth asking if that's a worthy goal, as these are really infrastructure logs and they "should" never have to worry about them. The app's own access logs are more likely to be relevant. But as a developer I do like to be able to see end to end. If we want to provide access logs from the router, we want them to be distinguishable from the rest of the router logs and available only to the right users. Access to aggregated user logs in ES is controlled based on which projects the user can see. Accordingly, logs are indexed by project, because access control in ES is available at the index level. To get access logs indexed correctly at load time, the project information has to be determined for each log message. In theory, something in the router pod could directly load logs to ES with the right metadata. While this might seem simplest I really doubt we will want that level of coupling between the router and logging solution. Probably we want to direct the logs somewhere and have the logging solution take it from there. Currently fluentd determines the relevant project based on which container the logs come from. That doesn't apply here as the source won't be a user app container. A new fluentd input method will need to be defined for this, to get the log entries wherever they are, determine whatever additional metadata is necessary (minimally, project), and include them in the log loading. A few possible integration points come to mind: 1. Write access logs to a log file in the pod. Create a sidecar that reads this log file to stdout so it can be aggregated and indexed like other container logs. Problem with this is that I'm not sure how fluentd could single out this particular container for different treatment (namely, has to look at the individual messages to determine what index they go to, instead of assigning them all to the originating pod). It may be possible based on router pod annotations or other metadata. 2. Write access logs to a syslog endpoint. I'm not sure what this endpoint would be - the node it happens to be on? That would assume we're comfortable with configuring a syslog endpoint on each node and can secure it so only the router can log to it. Perhaps logs could be directed to the fluentd instance running on the node, but then the router must be configured with that endpoint (not sure how; perhaps fluentd is listening on a host port?) and it would need to be secured. Somewhere else that fluentd can eventually access and read? Preferably, whatever we do could be re-used by other router solutions. I think the fluentd syslog endpoint has the most legs for that. I don't have the final answer - hopefully this helps clarify the next steps that need to be determined.
PR https://github.com/openshift/origin/pull/20260 adds the ability to send the access log to STDERR. We need to work out what next needs to happen to take that and make it available to the logging stack.