Bug 1751783

Summary: Output missing from commands with Flooded output
Product: Red Hat Enterprise Linux 8 Reporter: jstephen
Component: cockpitAssignee: Martin Pitt <mpitt>
Status: CLOSED ERRATA QA Contact: Jan Ščotka <jscotka>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 8.1CC: dpal, lcaparel, mmarusak, mpitt, omoris
Target Milestone: rc   
Target Release: 8.3   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-11-04 01:53:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1694846, 1793594    

Description jstephen 2019-09-12 15:00:59 UTC
Description of problem:
cockpit-session-recording uses the journal.journalctl method to retrieve a list of journal entries, process them, then present a list of Recordings in the UI. Intermittently(about 50% of the time), only a subset of the total entries are returned to the .stream(data) handler. When this happens the .done() handler never is called. 

The code we use:

https://github.com/Scribery/cockpit-session-recording/blob/master/src/recordings.jsx#L323

More details provided at the upstream issue: https://github.com/cockpit-project/cockpit/issues/12580

This leads to cockpit-session-recording not playing back all of the recorded output. For example, a recording containing the command 'cat /usr/share/dict/linux.words', the journalctl method may only retrieve all words up to 'm', or 'n', or 'r'

I believe this is also evident in the cockpit Terminal page, running a command such as:

   # find /

This is related to BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1694846

Version-Release number of selected component (if applicable):
Cockpit version: 199, 193
OS: Fedora 29, Fedora 30, RHEL8

How reproducible:
Often

Steps to Reproduce:
- dnf install tlog

- tlog-rec -w journal cat /usr/share/dict/linux.words

- In cockpit module code:

 let cmd = ["journalctl", "-q", "--lines=all", "_COMM=tlog-rec", "--output=json"];
 cockpit.spawn(cmd)
         .stream((data) => {
             console.log(data);   /* If data contains Zyzzogeton, all entries were returned */
             if (data.includes("Zyzzogeton")) {
                 console.log('Found string');
             }
         })
         .done(() => {
             console.log('Done!');
         });

- Can also be reproduced by installing cockpit-session-recording, and attempting to playback a recording with flooded output(cat /usr/share/dict/linux.words)

Actual Output:
Intermittently, not all of the output will be provided in .stream(data) and the .done() handler is never called

Comment 1 jstephen 2019-09-12 15:43:13 UTC
Hi Martin,

Sorry if it was not clear, but this is intentionally assigned to the 'cockpit' component(created by me) as we are having problems with the cockpit journalctl API.

Small correction from my previous comment, the line of code where we are having issues is here: https://github.com/Scribery/cockpit-session-recording/blob/master/src/recordings.jsx#L788

Comment 4 jstephen 2020-02-13 15:19:51 UTC
Hi Martin, I just wanted to follow-up on this to make sure it is not lost or forgotten. Let me know if you need any further information from me.

Comment 5 Martin Pitt 2020-02-14 11:23:24 UTC
@jstephen: Ack, it's not lost, just that this is a rather tricky issue and we haven't gotten around to this yet; sorry!

Comment 8 Martin Pitt 2020-05-11 14:02:17 UTC
I have been debugging this for a while now, and will continue to work on it. It will still take some time though. I keep updating the upstream issue with some details.

Comment 10 Martin Pitt 2020-05-12 16:07:33 UTC
I have a fix now: https://github.com/cockpit-project/cockpit/pull/14079 . It will still take a bit of cranking to land, but it's safe for 8.3.

Comment 17 errata-xmlrpc 2020-11-04 01:53:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (cockpit bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4511