Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: I would like to discuss possibility of adding stdout / stderr into xml journal created by rlJournalPrint. It could make journal usefull for investigating test execution details with additionals tools (like colorizers, diff tools, etc). Are there any major drawbacks? Version-Release number of selected component (if applicable): beakerlib-1.5-1.el6eso.noarch How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: e.g.: <test message="Checking for the presence of sos rpm">PASS</test> <test message="Running 'sosreport -o general --batch --verbose --no-multithread 2>&1 | tee no-multithread.out' (Expected 0, got 2)">FAIL</test> Expected results: could be: <test message="Checking for the presence of sos rpm"> <output fd="stdout">sos-2.2-29.el6.noarch</output> PASS </test> <test message="Running 'sosreport -o general --batch --verbose --no-multithread 2>&1 | tee no-multithread.out' (Expected 0, got 2)"> <output fd="stderr"> Usage: sosreport [options] sosreport: error: no such option: --no-multithread </output> FAIL </test> Additional info: Example above is simplified - command outputs may be arbitrary data so should be handled in xml accordingly.
Not sure now if this is really what we want (it would mean enlarging the journals by a crapload of most of the time useless inputs). I'll probably start some initiative for defining the NG journal, current single-file model ceased to be useful. It would be best to solve the issue systematically (define what data are needed when, where to put them, who are the consumers, etc.)
*** Bug 478967 has been marked as a duplicate of this bug. ***
My idea is the following: Not all "tests" in the journal are associated with a command. Usually there is just a check if a given condition holds. Usually we are interested in a witness, too: if something should be equal and it is not, we probably want to see the non-equivalent values. There are different witnesses for different checks. So I was thinking about separating the command (usually coming from rlRun invocations) from the rest of the tests. It would have a full-fledged XML record with additional information: - what was actually executed (different RFE) - how long did it take - how did it end - how it was expected to end - STDERR/STDOUT With outputs, things get complicated a bit: some commands produce mega/gigabytes of output (I know this, as I *had* such tests). We probably do not want to store always and everything. We probably want to store more when things end in unexpected way. But even then, I think we do not necessarily want to store whole output, always. I think the really huge outputs should be explicit opt-in. My idea is to have few environment variable-based "profiles", which controls how much content is stored in XML. These would be set using workflows for Beaker and env variables for local execution. For command outputs only, it could be something like this: none: no output is stored default: no output is stored when things pass, last 10 or 20 lines from both stdout/stderr is stored when they don't full: everything is stored Or even something smart like "capture all lines containing 'error' or 'warning'" That would allow different amount of information in test executions for various purposes. When someone is really interested only in the result, and not interested in any investigation, no output is stored (and everything probably runs a bit faster, because we redirect everything to /dev/null). Standard automation runs would produce little or no output. When we investigate a failing test, we could order an execution to capture everything. I can imagine e.g. tier tests being launched without any output captured to save time and save resources. As soon as a test would fail, a full output-captured run would be spawned for investigation. And so on. Thoughts?
This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component.
Is it really necessary to have it in journal? I think TESTOUT.log is sufficient for STDOUT and STDERR. I have implemented something like this in one of wireshark tests where it is useful to have about 2000 lines of the output in case of FAIL. I wonder if it is worth of implementing if the requirements differ a lot.
(In reply to Dalibor Pospíšil from comment #8) > Is it really necessary to have it in journal? I think TESTOUT.log is > sufficient for STDOUT and STDERR. > I have implemented something like this in one of wireshark tests where it is > useful to have about 2000 lines of the output in case of FAIL. I wonder if > it is worth of implementing if the requirements differ a lot. Consider eg. test result comparison, kind of diff. When one assert fails and the same assert in other test run pass, I would like to see the difference. But not only eg. command return code difference, but also stdout+stderr of these asserts. You can do it directly from TESTOUT.log, but it is: - always harder to correctly parse then xml journals - sometimes messed by tests (I have seen for example test printing 'beakerlib-like' PASSes into TESTOUT.log, which would confuse the parser) So for some tools like result differ/comparator with detailed outputs, it is not strictly necessary, but very useful and wanted.
Having said that and keeping journal speed in mind, I am open to discuss better ways. One idea which came to my mind is: we can probably just store pointers to TESTOUT.log in journal (start/end position). That would mean minimal journal overhead and still help tooling as discussed above.
If this should be implemented I would probably pick the pointer-like version. Actually I think the journal sometimes already contain unnecessary information, e.g. I would even omit debug messages due to the speed. Also I do not know much tools which would benefit from such detailed journal. I, personally, always go for TESTOUT.log AFAIK, this does not block anything or anyone I am setting priority low.