Bug 1527918

Summary: Application data logged in /var/log/messages by docker daemon
Product: Red Hat Enterprise Linux 7 Reporter: Jaspreet Kaur <jkaur>
Component: dockerAssignee: Daniel Walsh <dwalsh>
Status: CLOSED NOTABUG QA Contact: atomic-bugs <atomic-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 7.3CC: amurdaca, dornelas, dwalsh, lsm5, nbhatt, rhowe, sgaikwad
Target Milestone: rcKeywords: Extras
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-07-16 16:21:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1186913    

Description Jaspreet Kaur 2017-12-20 13:24:31 UTC
Description of problem:

It is seen that suddenly /var/log/messages getting filled up too fast and the messages were logged by dockerd-current

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results: /var/log/messages filling becuase application data is being written by docker daemon


Expected results: Application data should not be written.


Additional info:

Comment 5 Daniel Walsh 2017-12-23 10:58:52 UTC
Antonio any ideas here?  Is the user using the journal back end. Does switchin to using the json file back end help with the problem?

Is the user seeing the output of the containers in the syslog files?

Comment 8 Ryan Howe 2018-01-04 17:05:21 UTC
Journal will save to /var/log/messags if you have started and configured rsyslog  service to run with the imjournal module loaded. 

This seems like its just a case where rsyslog needs to be configured. 

I am not an expert with rsyslog but this issue looks like all that is needed is configuration to filter logs rsyslog writes when reading from the systemd-journal 


Example add the following to /etc/rsyslog.conf

```

if $programname == "dockerd-current" then /var/log/dockerd
& stop

if $programname == "journal" then /var/log/journal-test
& stop

```

# systemctl restart rsyslog 
# touch /var/log/journal /var/log/dockerd


  I really do no see a bug here, everything is working as expected. I guess the question can be asked, as to why logs from a container get the programname of journal and not docker or the container-name.

 I see container logs getting written to rsyslog as follows: 

Jan  4 12:00:44 infra-0 journal: 10.130.0.1 - - [04/Jan/2018:17:00:44 +0000] "GET /healthz HTTP/2.0" 200 0 "" "Go-http-client/2.0"

 The journalctl short output looks like this: 

Dec 31 15:48:20 example.test.com dockerd-current[25772]: 10.130.0.1 - - [31/Dec/2017:20:48:20 +0000] "GET /healthz HTTP/2.0" 200 0 "" "Go-http-client/2.0"


Further information:
https://docs.docker.com/v1.12/engine/admin/logging/journald/

Journald is saving information on where the logs are coming from, its just when you run `journalctl -u docker` you get the default output for journalctl which is `-o short`.

You can see more information if you run:

  # journalctl -u docker.service -o verbose
  
Or filter based on a single container:
  #  journalctl CONTAINER_NAME=k8s_registry_docker-registry-1-vbt77_default_83c43d27-dec2-11e7-a585-fa163efb7302_0
  or
  # journalctl CONTAINER_ID=77842a5ce7ea

Comment 12 Daniel Walsh 2018-02-09 13:30:52 UTC
I don't see rsyslog as a work around.  The messages are logged to journal and the journal works gives the information to rsyslog that it asks for.   If the customer does not want rsyslog recording the information then they need to tell rsyslog to not record it.

Comment 13 Derrick Ornelas 2018-05-08 20:14:57 UTC
(In reply to Daniel Walsh from comment #12)
> I don't see rsyslog as a work around.  The messages are logged to journal
> and the journal works gives the information to rsyslog that it asks for.  
> If the customer does not want rsyslog recording the information then they
> need to tell rsyslog to not record it.

It sounds like everything is working as expected when using the journald driver, correct, Dan?  

If I understand correctly they should adjust their docker log opt tags and configure rsyslog accordingly.

Comment 14 Daniel Walsh 2018-07-14 09:55:21 UTC
Yes.