Bug 1236664
| Summary: | Logs are duplicated in journald and /var/log/messages | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Itamar Turner-Trauring <itamar> |
| Component: | systemd | Assignee: | Alec Benson <albenson> |
| Status: | CLOSED WONTFIX | QA Contact: | qe-baseos-daemons |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.3 | CC: | dwalsh, itamar, jrieden, lnykryn, lsm5, systemd-maint-list |
| Target Milestone: | rc | Keywords: | Extras |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2015-07-16 13:42:27 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Itamar Turner-Trauring
2015-06-29 16:48:52 UTC
Docker by default does not log anything to journald or syslog, (We are working to fix this.) The only output that ends up in the journal is stdout/stderr that is being recorded in the docker.service unit file. Does this show up in /var/log/messages? Yes, e.g.: [root@ip-10-239-173-248 ~]# tail -F /var/log/messages Jun 29 16:09:28 ip-10-239-173-248 docker: time="2015-06-29T16:09:28Z" level=info msg="-job image_inspect(7bf0ec35adaff1648db2073771756d92739fd6cddec5d8a080ac64f4f73f14e3) = OK (0)" Jun 29 16:09:28 ip-10-239-173-248 docker: time="2015-06-29T16:09:28Z" level=info msg="GET /v1.15/containers/2fbe844fc9a01eab0451e9730e5e651b0455c4bed31085d7fcf0d821b0d7c8f6/json" Jun 29 16:09:28 ip-10-239-173-248 docker: time="2015-06-29T16:09:28Z" level=info msg="+job container_inspect(2fbe844fc9a01eab0451e9730e5e651b0455c4bed31085d7fcf0d821b0d7c8f6)" Jun 29 16:09:28 ip-10-239-173-248 docker: time="2015-06-29T16:09:28Z" level=info msg="-job container_inspect(2fbe844fc9a01eab0451e9730e5e651b0455c4bed31085d7fcf0d821b0d7c8f6) = OK (0)" Jun 29 16:09:28 ip-10-239-173-248 docker: time="2015-06-29T16:09:28Z" level=info msg="GET /v1.15/images/7bf0ec35adaff1648db2073771756d92739fd6cddec5d8a080ac64f4f73f14e3/json" Basically every interaction with the Docker API ends up leaving log messages, and so applications that do a lot of polling (which is unfortunately the way we currently do things) spew lots of logs into journald and therefore into /var/log/messages. (Spewing lots of logs into journald is not really an issue because it has the max%-of-filesystem-space option by default.) Alec could you look into this. I am having trouble reproducing this. I tried following the steps that you suggested in the original post and am not seeing any of the messages that you have reported seeing appear in /var/log/messages. Sorry, I will try to get more accurate reproducing instructions. No problem. In the meantime, this may solve your problem: in /etc/systemd/journald.conf, add the key ForwardToSyslog=no Hi, just wanted to check in -- did the above solve the issue? Or are you still seeing this problem? Here's an example that worked for me; conceivably it's CentOS only but my understanding is that they built these RPMs straight off the RHEL7 Extras packages. Perhaps I'm wrong. # cat /etc/redhat-release CentOS Linux release 7.0.1406 (Core) # tail -1000 /var/log/messages | grep docker Jul 14 18:20:55 localhost docker: time="2015-07-14T18:20:55Z" level=info msg="GET /v1.18/containers/json" Jul 14 18:20:55 localhost docker: time="2015-07-14T18:20:55Z" level=info msg="+job containers()" Jul 14 18:20:55 localhost docker: time="2015-07-14T18:20:55Z" level=info msg="-job containers() = OK (0)" # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f38bccb858b3 busybox:latest "\"sh -c 'echo -n '# About an hour ago Up About an hour 0.0.0.0:35813->9000/tcp flocker--flocker_acceptance_test_api_ContainerAPITests_test_linking-390822 # tail -1000 /var/log/messages | grep docker Jul 14 18:20:55 localhost docker: time="2015-07-14T18:20:55Z" level=info msg="GET /v1.18/containers/json" Jul 14 18:20:55 localhost docker: time="2015-07-14T18:20:55Z" level=info msg="+job containers()" Jul 14 18:20:55 localhost docker: time="2015-07-14T18:20:55Z" level=info msg="-job containers() = OK (0)" Jul 14 18:21:26 localhost docker: time="2015-07-14T18:21:26Z" level=info msg="GET /v1.18/containers/json" Jul 14 18:21:26 localhost docker: time="2015-07-14T18:21:26Z" level=info msg="+job containers()" Jul 14 18:21:26 localhost docker: time="2015-07-14T18:21:26Z" level=info msg="-job containers() = OK (0)" # rpm -qi docker Name : docker Version : 1.6.2 Release : 14.el7.centos ... Source RPM : docker-1.6.2-14.el7.centos.src.rpm ... It looks like you have stdout from your docker.service unit file redirecting to syslog. This information is not being logged by docker directly, to the best of my knowledge docker is not calling syslog. As the original description explains, logs (stdout in this case) from docker goes to journald, and journald by default passes everything to rsyslog. To put it another way: any service that relies on journald for logging is getting two copies of its logs written out, one to journald and then one passed on by journald to rsyslog and therefore to /var/log/messages by default. So one could argue this is a journald issue. Did you try adding ForwardToSyslog=no to /etc/systemd/journald.conf? I believe that should take care of the issue you are describing. I will move this to journald/systemd, and I will guess they will close it down as won't fix. In rhel7 we have journal in non-persistent mode, so on disk logs are only in /var/log/messages. But you can make it persistent and disable rsyslog. ForwardToSyslog=no will not help since rsyslog is reading journal data directly. The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days |