Bug 1691354

Summary:	Using tlog makes tailing logs unreadable
Product:	Red Hat Enterprise Linux 8	Reporter:	Ben Breard <bbreard>
Component:	tlog	Assignee:	Nikolai Kondrashov <nikolai.kondrashov>
Status:	CLOSED DUPLICATE	QA Contact:	ipa-qe <ipa-qe>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	8.0	CC:	jstephen, lmanasko, mthacker, nikolai.kondrashov
Target Milestone:	rc
Target Release:	8.0
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-07-03 15:24:17 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Ben Breard 2019-03-21 13:23:07 UTC

Description of problem:

While using a session that's being recorded with tlog and tailing (-f) the journal or /var/log/messages, the messages loop in the log while it's recording your session. This is an incredibly common administrative and troubleshooting task and tlog makes it more difficult. Filtering the journal properly or tailing an application log like apache doesn't have this problem, but I suspect a lot of users don't always do this.

Version-Release number of selected component (if applicable):
8.0 HTB

tlog-5-1.el8

How reproducible:
100%

Steps to Reproduce:
1. yum -y install tlog cockpit-session-recording cockpit-ws
2. systemctl start cockpit.socket
3. Access the session recording page in cockpit, accept the defaults for configuration
4. in a recorded session, I was using ssh, run: journalctl -f


Actual results:
Wait about ~10 seconds and then it gets pretty crazy w/ garbage streaming by insanely fast.

Expected results:
tail normal log output.

Additional info:
There are a number of configuration options available and likely that using other writers or settings that this behavior can be changed. This seems like such a common use case that we would want the defaults to work w/ basic log tailing.

Comment 1 Mark Thacker 2019-03-21 13:29:16 UTC

I have seen this issue as well. Without any other filtering parameters, the user is seeing the act of recording the results of viewing the system journal or /var/log/messages. Since viewing generates, which then print to the screen, this causes session recording to record this action, which generates more records, etc.... 

At the very least we need to provide a release note and documentation about how to invoke system journal viewing or filter a search request to avoid seeing the session recording activity while viewing the journal.

Comment 2 Nikolai Kondrashov 2019-03-25 09:47:42 UTC

Yes, documenting the behavior as well as a way to filter tlog messages out would help. Another thing which could be done to mitigate this is to rate-limit (throttle) the terminal throughput within tlog, so at least the logs are not flooded and the system is not overwhelmed.

Comment 3 jstephen 2019-03-27 18:18:20 UTC

For documentation purpose, this may work:

    # journalctl -f | grep -v 'tlog-rec-session'

I could not find a journalctl command to exclude certain messages by field/unit/etc. I believe rsyslog could also be configured to send tlog-rec-session logs to a different log file.

As tlog-rec-session is logging the terminal I/O to the journal or syslog by design, I don't really see a fix for this on the tlog side. Only the workarounds mentioned here and use of rate-limiting.

Comment 4 Ben Breard 2019-04-16 21:03:36 UTC

I think rate limiting and documenting this behavior is a great first step. I hit this instantly when trying out tlog so it will be interesting to see how many customers run into this.

Comment 5 jstephen 2019-05-08 17:48:58 UTC

Hi Lucie,

Could you please assist with documenting this behaviour to mitigate customer concern/support cases?

Comment 7 jstephen 2019-05-10 13:53:47 UTC

Documentation BZ created below:

https://bugzilla.redhat.com/show_bug.cgi?id=1708662

Comment 8 jstephen 2019-07-03 15:24:17 UTC


*** This bug has been marked as a duplicate of bug 1708662 ***