This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1252648 - [RFE] Support for the way of docker container log management
[RFE] Support for the way of docker container log management
Status: CLOSED CURRENTRELEASE
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging (Show other bugs)
3.0.0
Unspecified Unspecified
unspecified Severity low
: ---
: ---
Assigned To: Luke Meyer
Anping Li
:
Depends On:
Blocks: 1267746
  Show dependency treegraph
 
Reported: 2015-08-11 20:53 EDT by Kenjiro Nakayama
Modified: 2015-12-07 13:04 EST (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-01 03:28:48 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2072853 None None None Never

  None (edit)
Description Kenjiro Nakayama 2015-08-11 20:53:11 EDT
Description of problem:

After creating docker container, the size of the container's log in /var/lib/docker/containers/HASH/HASH-json.log is increasing. The issue is that we don't have any way to remove / tidy / rotate / set max size of the log. We have to destroy container to remove the log.


Steps to Reproduce:
1.
2.
3.


Actual results:

During running the container, the container's log(/var/lib/docker/containers/HASH/HASH-json.log) would be increasing.

Expected results / Additional info::

We would like to have some command to manage the log.
Docker upstream has already merged the patch to solve this issue : https://github.com/docker/docker/pull/11485


Additional:

I know we may be able to manage it by log aaggregation outside of individual docker containers[1]. However, we should be able to manage it, without log aggregation.

[1] https://docs.openshift.com/enterprise/3.0/admin_guide/aggregate_logging.html
Comment 2 Luke Meyer 2015-08-13 08:41:28 EDT
Actually log aggregation doesn't yet and probably won't deal with rotating the docker logs under the json-file logging driver. With docker-1.7 the driver will provide rotation options that should solve this. For now, your only recourse is:

1) "rotate" them separately, either with logrotate or similar, or just plain truncating them.
2) change the docker logging driver to send them to syslog; unfortunately this means you'll get nothing from "docker logs" or "oc logs", and I don't think the logs are adequately identified in the syslog such that you could narrow them down to a specific container/pod, so it's not a very useful option.

I'm not certain when docker 1.7 is coming out but I don't think it's far away.
Comment 3 Kenjiro Nakayama 2015-08-13 09:42:43 EDT
> With docker-1.7 the driver will provide rotation options that should solve this. For now, your only recourse is:

Current OpenShift(v3.0.1.0) includes docker-1.7.1. So you mean 1.8? 

[root@ose3-master ~]# docker version
Client version: 1.7.1
Client API version: 1.19
Package Version (client): docker-1.7.1-108.el7.x86_64

The fix in the upstream also says that milestone is "1.8.0": https://github.com/docker/docker/pull/11485
Comment 4 Luke Meyer 2015-08-14 11:38:50 EDT
(In reply to Kenjiro Nakayama from comment #3)
> Current OpenShift(v3.0.1.0) includes docker-1.7.1. So you mean 1.8? 

I expected it was coming out soon, didn't realize it was already out :)

> The fix in the upstream also says that milestone is "1.8.0":
> https://github.com/docker/docker/pull/11485

It seems you're right; I saw there were new logging features coming out in 1.7 (e.g. the journald driver) and saw the source code docs for rotation and assumed they were together. But I just tested adding the log rotation parameters and they did nothing.

So, I guess we can expect that with docker 1.8 or a backport; in the meantime, the two options I gave seem to be what's available. We will have to evaluate whether our log aggregation solution can help manage log rotation in the short term.
Comment 5 Ryan Howe 2015-09-22 10:41:02 EDT
Would this log parameter be something that we will be able to define in the build or deployment config if we use docker 1.8?
Comment 6 Ricardo Martinelli de Oliveira 2015-09-24 18:00:55 EDT
Could you please check Ryan comment and let us know how the log parameter will be defined?
Comment 9 Luke Meyer 2015-10-12 07:21:23 EDT
Thought I already responded to this, somehow must have failed.

See the following log options which should be available with Docker 1.8:
https://docs.docker.com/reference/logging/overview/#the-json-file-options

It's worth noting that while we will default to json-file driver for the next release, there is a journald driver available which also works with "docker logs" and thus "oc logs". I haven't investigated to see if this actually works under kubernetes (there could be e.g. formatting differences that require attention) or how log aggregation would be different, but it's promising.
Comment 19 Luke Meyer 2015-11-30 11:18:44 EST
With Docker 1.8 now shipping, log rotation with the json-file driver is just some options in Docker. What remains is to make customers aware of this via docs, and preferably to enable them to be set automatically during an ansible install. It may be appropriate to spawn two RFEs from this and close this CURRENTRELEASE.
Comment 21 Kenjiro Nakayama 2015-11-30 22:26:02 EST
(In reply to Luke Meyer from comment #19)

I opened these two bugzilla:

https://bugzilla.redhat.com/show_bug.cgi?id=1286895
https://bugzilla.redhat.com/show_bug.cgi?id=1286893

Please feel free to close this.

Note You need to log in before you can comment on or make changes to this bug.