Bug 1252648

Summary:	[RFE] Support for the way of docker container log management
Product:	OpenShift Container Platform	Reporter:	Kenjiro Nakayama <knakayam>
Component:	Logging	Assignee:	Luke Meyer <lmeyer>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Anping Li <anli>
Severity:	low	Docs Contact:
Priority:	unspecified
Version:	3.0.0	CC:	asolanas, dmcphers, ederevea, erich, knakayam, libra-bugs, lmeyer, nicholas_schuetz, pep, philfest, rhowe, rmartine
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-12-01 08:28:48 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1267746

Description Kenjiro Nakayama 2015-08-12 00:53:11 UTC

Description of problem:

After creating docker container, the size of the container's log in /var/lib/docker/containers/HASH/HASH-json.log is increasing. The issue is that we don't have any way to remove / tidy / rotate / set max size of the log. We have to destroy container to remove the log.


Steps to Reproduce:
1.
2.
3.


Actual results:

During running the container, the container's log(/var/lib/docker/containers/HASH/HASH-json.log) would be increasing.

Expected results / Additional info::

We would like to have some command to manage the log.
Docker upstream has already merged the patch to solve this issue : https://github.com/docker/docker/pull/11485


Additional:

I know we may be able to manage it by log aaggregation outside of individual docker containers[1]. However, we should be able to manage it, without log aggregation.

[1] https://docs.openshift.com/enterprise/3.0/admin_guide/aggregate_logging.html

Comment 2 Luke Meyer 2015-08-13 12:41:28 UTC

Actually log aggregation doesn't yet and probably won't deal with rotating the docker logs under the json-file logging driver. With docker-1.7 the driver will provide rotation options that should solve this. For now, your only recourse is:

1) "rotate" them separately, either with logrotate or similar, or just plain truncating them.
2) change the docker logging driver to send them to syslog; unfortunately this means you'll get nothing from "docker logs" or "oc logs", and I don't think the logs are adequately identified in the syslog such that you could narrow them down to a specific container/pod, so it's not a very useful option.

I'm not certain when docker 1.7 is coming out but I don't think it's far away.

Comment 3 Kenjiro Nakayama 2015-08-13 13:42:43 UTC

> With docker-1.7 the driver will provide rotation options that should solve this. For now, your only recourse is:

Current OpenShift(v3.0.1.0) includes docker-1.7.1. So you mean 1.8? 

[root@ose3-master ~]# docker version
Client version: 1.7.1
Client API version: 1.19
Package Version (client): docker-1.7.1-108.el7.x86_64

The fix in the upstream also says that milestone is "1.8.0": https://github.com/docker/docker/pull/11485

Comment 4 Luke Meyer 2015-08-14 15:38:50 UTC

(In reply to Kenjiro Nakayama from comment #3)
> Current OpenShift(v3.0.1.0) includes docker-1.7.1. So you mean 1.8? 

I expected it was coming out soon, didn't realize it was already out :)

> The fix in the upstream also says that milestone is "1.8.0":
> https://github.com/docker/docker/pull/11485

It seems you're right; I saw there were new logging features coming out in 1.7 (e.g. the journald driver) and saw the source code docs for rotation and assumed they were together. But I just tested adding the log rotation parameters and they did nothing.

So, I guess we can expect that with docker 1.8 or a backport; in the meantime, the two options I gave seem to be what's available. We will have to evaluate whether our log aggregation solution can help manage log rotation in the short term.

Comment 5 Ryan Howe 2015-09-22 14:41:02 UTC

Would this log parameter be something that we will be able to define in the build or deployment config if we use docker 1.8?

Comment 6 Ricardo Martinelli de Oliveira 2015-09-24 22:00:55 UTC

Could you please check Ryan comment and let us know how the log parameter will be defined?

Comment 9 Luke Meyer 2015-10-12 11:21:23 UTC

Thought I already responded to this, somehow must have failed.

See the following log options which should be available with Docker 1.8:
https://docs.docker.com/reference/logging/overview/#the-json-file-options

It's worth noting that while we will default to json-file driver for the next release, there is a journald driver available which also works with "docker logs" and thus "oc logs". I haven't investigated to see if this actually works under kubernetes (there could be e.g. formatting differences that require attention) or how log aggregation would be different, but it's promising.

Comment 19 Luke Meyer 2015-11-30 16:18:44 UTC

With Docker 1.8 now shipping, log rotation with the json-file driver is just some options in Docker. What remains is to make customers aware of this via docs, and preferably to enable them to be set automatically during an ansible install. It may be appropriate to spawn two RFEs from this and close this CURRENTRELEASE.

Comment 21 Kenjiro Nakayama 2015-12-01 03:26:02 UTC

(In reply to Luke Meyer from comment #19)

I opened these two bugzilla:

https://bugzilla.redhat.com/show_bug.cgi?id=1286895
https://bugzilla.redhat.com/show_bug.cgi?id=1286893

Please feel free to close this.