RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1309491 - stderr from script appears as stdout
Summary: stderr from script appears as stdout
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: openscap
Version: 6.8
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Jan Černý
QA Contact: Marek Haicman
URL:
Whiteboard:
Depends On:
Blocks: 1335121 1392901 1393496
TreeView+ depends on / blocked
 
Reported: 2016-02-18 00:10 UTC by Alois Mahdal
Modified: 2017-03-21 11:31 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-03-21 11:31:39 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
old-style-result.html (255.93 KB, text/html)
2016-11-08 12:00 UTC, Petr Stodulka
no flags Details
new-style-fixed-result.html (636.14 KB, text/html)
2016-11-08 13:37 UTC, Petr Stodulka
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:0754 0 normal SHIPPED_LIVE openscap bug fix and enhancement update 2017-03-21 12:46:08 UTC

Description Alois Mahdal 2016-02-18 00:10:10 UTC
Description of problem
======================

Errors from module that must have been printed to stderr are
found in stdout XML node.

I have only found this by accident with `mysql_configuration_changes`
where the errors come from syntax errors and file-not-found errors(!),
but since there's no redirection, I'm assuming the output is merged on
PA level.

Relevant XML node from mentioned module looks like this:

    <ns0:rule-result idref="xccdf_preupg_rule_databases_mysql_configuration_changes_configuration" time="2016-02-17T18:39:28" weight="1.000000">
          <ns0:result>needs_action</ns0:result>
          <ns0:check system="http://open-scap.org/page/SCE">
            <ns0:check-import import-name="stdout">/usr/share/preupgrade/common.sh: line 226: syntax error near unexpected token `}'
    /usr/share/preupgrade/common.sh: line 226: `}'
    /root/preupgrade/RHEL6_7/databases/mysql/configuration_changes/configuration.sh: line 24: check_rpm_to: command not found
    /root/preupgrade/RHEL6_7/databases/mysql/configuration_changes/configuration.sh: line 29: check_root: command not found
    /root/preupgrade/RHEL6_7/databases/mysql/configuration_changes/configuration.sh: line 31: ../mysql-common.sh: No such file or directory
    /root/preupgrade/RHEL6_7/databases/mysql/configuration_changes/configuration.sh: line 57: backup_config: command not found
    INPLACERISK: MEDIUM: /etc/my.cnf: option 'plugin-load=innodb=' is not supported
    sed: can't read /root/preupgrade//etc/my.cnf: No such file or directory
    INPLACERISK: HIGH: /etc/my.cnf: option 'innodb_file_io_threads' is not supported
    sed: can't read /root/preupgrade//etc/my.cnf: No such file or directory
    INPLACERISK: HIGH: /etc/my.cnf: option 'language' is deprecated
    sed: can't read /root/preupgrade//etc/my.cnf: No such file or directory
    /root/preupgrade/RHEL6_7/databases/mysql/configuration_changes/configuration.sh: line 103: : No such file or directory
    INPLACERISK: HIGH: /etc/my.cnf: option 'log-bin-trust-routine-creators' was removed in MariaDB 5.5
    sed: can't read /root/preupgrade//etc/my.cnf: No such file or directory
    /root/preupgrade/RHEL6_7/databases/mysql/configuration_changes/configuration.sh: line 103: : No such file or directory
    INPLACERISK: HIGH: /etc/my.cnf: option 'table_lock_wait_timeout' was removed in MariaDB 5.5
    sed: can't read /root/preupgrade//etc/my.cnf: No such file or directory

    The files mentioned above are backups of your original configuration files.
    After the upgrade you must manually replace the contents of the default
    MariaDB 5.5 configuration (as is appropriate to your environment), with the
    information provided about replaced or deprecated options listed in the
    backups.  You must restart MariaDB for these changes to take effect.

    </ns0:check-import>
            <ns0:check-import import-name="stderr">
                </ns0:check-import>
            <ns0:check-export export-name="RESULT_PART" value-id="xccdf_preupg_value_databases_mysql_configuration_changes_configuration_state_result_part" />
            <ns0:check-export export-name="CURRENT_DIRECTORY" value-id="xccdf_preupg_value_databases_mysql_configuration_changes_configuration_state_current_directory" />
            <ns0:check-export export-name="SOLUTION_FILE" value-id="xccdf_preupg_value_databases_mysql_configuration_changes_configuration_state_solution_file" />
            <ns0:check-export export-name="REPORT_DIR" value-id="xccdf_preupg_value_report_dir" />
            <ns0:check-export export-name="UPGRADE" value-id="xccdf_preupg_value_upgrade" />
            <ns0:check-export export-name="MIGRATE" value-id="xccdf_preupg_value_migrate" />
            <ns0:check-export export-name="DIST_NATIVE" value-id="xccdf_preupg_value_dist_native" />
            <ns0:check-export export-name="DEVEL_MODE" value-id="xccdf_preupg_value_devel_mode" />
            <ns0:check-export export-name="TMP_PREUPGRADE" value-id="xccdf_preupg_value_tmp_preupgrade" />
            <ns0:check-content-ref href="databases/mysql/configuration_changes/configuration.sh" />
          </ns0:check>
        </ns0:rule-result>

As you can see, the syntax and sed errors are in `<ns0:check-import
import-name="stdout"/>` node, although they must have been printed to
stderr by relevant programs.  At the same time, `stderr` node is empty.


Version-Release number of selected component
============================================

preupgrade-assistant-2.1.4-9.el6.noarch
preupgrade-assistant-contents-0.6.43-1.el6.noarch
preupgrade-assistant-contents-67-0.6.43-1.el6.noarch
preupgrade-assistant-staticdata-0.1.2-1.el6.noarch
preupgrade-assistant-staticdata-67-0.1.2-1.el6.noarch


How reproducible
================

Always


Steps to Reproduce
==================

 1. Have a module print something to stderr

 2. Look for the string in following nodes:

        <ns0:check-import import-name="stdout"/>
        <ns0:check-import import-name="stderr"/>


Actual results
==============

The string is found in stdout node but not in stderr node.


Expected results
================

The string should be found in stderr node but not in stdout node.


Additional info
===============

 *  This bug is not about the actual problems in the mysql-related module;
    separate bug will be recorded for that.

 *  As a consequence, this disables a routine test "no stderr printed"
    since from XML POV there's never any stderr.

 *  It would make sense for PA to raise error state just based on the
    presence of stderr.  I don't know if PA has such functionality (due
    to this bug we can't know); if not, let's open RFE for that...

Comment 2 Alois Mahdal 2016-02-18 02:15:36 UTC
(In reply to Alois Mahdal from comment #0)
[...]
> 
>  *  This bug is not about the actual problems in the mysql-related module;
>     separate bug will be recorded for that.

Turns out it's in common.sh; reported as bug 1309519.

Comment 3 Alois Mahdal 2016-03-03 18:48:04 UTC
@tcerna, this should easy to check once we have any module that prints to stderr. (I guess there will always be some, or we can create own test module eg. using the tool provided with PA).

Comment 4 Ondrej Vasik 2016-03-07 09:37:59 UTC
Petr, can you please take a look on this? It is very likely it may cause unexpected glitches in report.html...

Comment 5 Petr Hracek 2016-05-30 11:48:24 UTC
Sure. I will look on it.

Comment 6 Alois Mahdal 2016-05-30 20:16:01 UTC
Note to QA:

Functions

    preupgrade_assistant__get_stdout
    preupgrade_assistant__get_stderr

get the module's stderr and stdout.  The rule ID can be passed as (sole) argument or set as $preupgrade_assistant__rule.

Comment 7 Petr Hracek 2016-06-06 12:18:06 UTC
This has to be fixed in openscap first of all.
https://github.com/OpenSCAP/openscap/issues/423

This is not preupgrade-assistant issue.

Reassigning to openscap.

Comment 8 Jan Černý 2016-07-21 12:29:05 UTC
From what I understand from the Script Check Engine definition, https://www.open-scap.org/features/other-standards/sce/ the SCE does not have <check-import import-name="stderr"\> element. Therefore I think we are not able to fix this without a change in the specification.

Comment 9 Alois Mahdal 2016-07-21 12:57:48 UTC
Thanks for info.

This is quite limiting for us in testing of in-place upgrades; it makes it hard to catch unexpected script errors in automated tests.  (I imagine this would be problem for any other rule content testing.)

"Change in the specification" sounds a bit scary;  is it possible to estimate how "fast" this can be delivered?  Is it realistic to have it in 6.9 (or say, in all innocence, even z-stream)?

Or are we doomed to resort to hacks on our part?  (I can imagine wrapping every script but I can also imagine faces of in-place upgrade devels when proposing that, not sure which is more scary.)

Comment 10 Jan Černý 2016-07-22 06:52:00 UTC
I have discussed this bug with mpreisle.

We can add new elements into the specification, but we can't change the meaning of already existing to avoid breaking things.

I agree that check-import/@name="stdout" is very confusing name if it contains output from both stderr and stdout, but I don't see a way how to change it and keep the backward compatibility at the same time.

Therefore the suggestion is to add 2 new elements:

- check-import/@name="stderr"
- check-import/@name="stdout_only"

This change is realistic to deliver in RHEL 6.9.

Comment 11 Alois Mahdal 2016-07-22 09:52:17 UTC
Thank you for prompt reply!

(By the way, why not "stderr_only" + "stdout_only"?)

Comment 14 Martin Preisler 2016-10-13 16:17:11 UTC
Hi Petr,
I have prototyped the fix in https://github.com/OpenSCAP/openscap/pull/555 and would appreciate your comments. Does it work for your use-case?

We are reluctant to do these kind of changes in stable branches but I think in this case we could make an exception. If the rest of the team agrees.

Comment 15 Alois Mahdal 2016-11-07 17:30:45 UTC
(In reply to Martin Preisler from comment #14)
> Hi Petr,
> I have prototyped the fix in https://github.com/OpenSCAP/openscap/pull/555
> and would appreciate your comments. Does it work for your use-case?

Our use case was in downstream test suite.  The aim was to raise fail condition every time a script would produce any stderr, which was impossible as we had only the merged output available.  Your fix seems to solve that well.

It's worth mentioning, though, that since this was filed, we have changed the way the data is passed around (bug 1361378) in such way that we are not dependent on this change so much.

Comment 17 Petr Stodulka 2016-11-08 09:00:53 UTC
According to information on github and changelog, it is not part of the build. However, I guess it is not problem apply the patch manually and create own build for testing. I will look at it this week yet.

Comment 19 Petr Stodulka 2016-11-08 12:00:12 UTC
Created attachment 1218495 [details]
old-style-result.html

I found that the implemented feature broke our reports (old style report in attachment), because we used modified XSL files.
Don't know if another changes in PreupgradeAssistant will be needed. However, if this will be part of rhel, we have
to do modification on our side too.

- https://github.com/upgrades-migrations/preupgrade-assistant/tree/project_hierarchy_cleanup/data/html_report_complex
- https://github.com/upgrades-migrations/preupgrade-assistant/tree/project_hierarchy_cleanup/data/html_report_simple

Comment 20 Petr Stodulka 2016-11-08 13:37:48 UTC
Created attachment 1218524 [details]
new-style-fixed-result.html

I have applied changes to our XSL files derived from current upstream version. However XSL files for old report have to be modified yet. This will be done during week, after discussion with Martin.

According to results, it seems good to me. So +1 from my side for this patch.

Comment 23 Marek Haicman 2016-12-11 22:53:49 UTC
Thank you Petr for check on the preupgrade assistant side.

Confirming the fix is in version openscap-1.2.12-1.el6.x86_64

When using rule definition

  <Rule selected="true" id="xccdf_moc.elpmaxe.www_rule_1">
    <title>Test SCE Rule</title>
    <check system="http://open-scap.org/page/SCE">
      <check-import import-name="stdout" />
      <check-import import-name="stderr" />
      <check-content-ref href="test_output.sh"/>
    </check>
  </Rule>

where test_output prints "out" and "err" to respective channels, results are:

===
OLD (openscap-1.2.8-2.el6.x86_64):
    <rule-result idref="xccdf_moc.elpmaxe.www_rule_1" time="2016-12-11T17:43:27" weight="1.000000">
      <result>pass</result>
      <check system="http://open-scap.org/page/SCE">
        <check-import import-name="stdout">out
err
</check-import>
        <check-import import-name="stderr">
      </check-import>
        <check-content-ref href="test_output.sh"/>
      </check>
    </rule-result>

===
NEW:
    <rule-result idref="xccdf_moc.elpmaxe.www_rule_1" time="2016-12-11T17:42:08" weight="1.000000">
      <result>pass</result>
      <check system="http://open-scap.org/page/SCE">
        <check-import import-name="stdout">out
</check-import>
        <check-import import-name="stderr">err
</check-import>
        <check-content-ref href="test_output.sh"/>
      </check>
    </rule-result>

Comment 26 errata-xmlrpc 2017-03-21 11:31:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0754.html


Note You need to log in before you can comment on or make changes to this bug.