Bug 1317118 - upstream tests in test_oscap.py raise AssertionError
Summary: upstream tests in test_oscap.py raise AssertionError
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: preupgrade-assistant
Version: 6.8
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Petr Hracek
QA Contact: Alois Mahdal
URL:
Whiteboard:
Depends On:
Blocks: 1330043
TreeView+ depends on / blocked
 
Reported: 2016-03-12 05:19 UTC by Alois Mahdal
Modified: 2016-04-26 07:03 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-04-26 07:03:31 UTC
Target Upstream Version:

Attachments (Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Description Alois Mahdal 2016-03-12 05:19:42 UTC 
Description of problem
======================

Test failures in `tests/test_oscap.py`:

    :: [  BEGIN   ] :: Run all tests in file: ./tests/test_oscap.py :: actually running ' nosetests -vv ./tests/test_oscap.py'
    nose.config: INFO: Excluding tests matching ['test_compose|test_final_compose|generate_test_xml']
    Basic test for FAIL SCE ... FAIL
    Basic test for FIXED SCE ... FAIL
    Basic test for FAIL SCE ... FAIL
    Basic test for FAIL SCE ... FAIL
    Basic test for NOT_APPLICABLE SCE ... FAIL
    Basic test for PASS SCE ... FAIL

    ======================================================================
    FAIL: Basic test for FAIL SCE
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/tmp/tmp.PU6MoJgEVc/preupgrade-assistant/tests/test_oscap.py", line 169, in test_fail
        self.assertEqual(a.run_scan(), 2)
    AssertionError: 1 != 2

    ======================================================================
    FAIL: Basic test for FIXED SCE
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/tmp/tmp.PU6MoJgEVc/preupgrade-assistant/tests/test_oscap.py", line 273, in test_fixed
        self.assertEqual(a.run_scan(), 0)
    AssertionError: 1 != 0

    ======================================================================
    FAIL: Basic test for FAIL SCE
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/tmp/tmp.PU6MoJgEVc/preupgrade-assistant/tests/test_oscap.py", line 221, in test_needs_action
        self.assertEqual(a.run_scan(), 2)
    AssertionError: 1 != 2

    ======================================================================
    FAIL: Basic test for FAIL SCE
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/tmp/tmp.PU6MoJgEVc/preupgrade-assistant/tests/test_oscap.py", line 194, in test_needs_inspection
        self.assertEqual(a.run_scan(), 2)
    AssertionError: 1 != 2

    ======================================================================
    FAIL: Basic test for NOT_APPLICABLE SCE
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/tmp/tmp.PU6MoJgEVc/preupgrade-assistant/tests/test_oscap.py", line 248, in test_not_applicable
        self.assertEqual(a.run_scan(), 0)
    AssertionError: 1 != 0

    ======================================================================
    FAIL: Basic test for PASS SCE
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/tmp/tmp.PU6MoJgEVc/preupgrade-assistant/tests/test_oscap.py", line 141, in test_pass
        self.assertEqual(return_string, 0)
    AssertionError: 1 != 0



Version-Release number of selected component
============================================

preupgrade-assistant-2.1.5-3.el6


How reproducible
================

Always


Steps to Reproduce
==================

 1. Run /CoreOS/preupgrade-assistant/6to7/assistant/upstream
 2. Check for above errors in `tests/test_oscap.py`


Additional info
===============

There are more failing tests than this, but since there are several types
of errors; to allow for proper triaging I'm reporting them separately.
Comment 2 Alois Mahdal 2016-03-12 05:34:29 UTC 
Also, it looks like some tests are mislabeled:

 *  test_not_applicable() => "Basic test for NOT_APPLICABLE SCE",

 *  test_fail() => "Basic test for FAIL SCE"

but

 *  test_needs_action() => "Basic test for FAIL SCE".
Comment 4 Petr Hracek 2016-04-26 06:58:16 UTC 
This is fixed by upstream commit https://github.com/phracek/preupgrade-assistant/commit/5f45a20352c59cc130de9db02d75670a53104505#diff-3bf2b8112adbb9a459970ff22d4f1a66
Comment 5 Petr Hracek 2016-04-26 07:02:21 UTC 
I plan to use gitlab as a CI tests for preupgrade-assistant

running test
running egg_info
writing preupgrade_assistant.egg-info/PKG-INFO
writing top-level names to preupgrade_assistant.egg-info/top_level.txt
writing dependency_links to preupgrade_assistant.egg-info/dependency_links.txt
reading manifest file 'preupgrade_assistant.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'Makefile'
warning: no files found matching '*' under directory 'preup_creator'
writing manifest file 'preupgrade_assistant.egg-info/SOURCES.txt'
running build_ext
test_all (tests.test_preup.TestPreupg)
Basic test for whole program ... ok
test_migrate (tests.test_preup.TestPreupg)
Basic test for whole program ... ok
test_upgrade (tests.test_preup.TestPreupg)
Basic test for whole program ... ok
test_opts (tests.test_preup.TestCLI)
basic test of several options ... ok
test_hashes (tests.test_preup.TestHashes) ... ok
test_solution_bold_tag (tests.test_preup.TestSolutionReplacement) ... ok
test_solution_file_bold_tag (tests.test_preup.TestSolutionReplacement) ... ok
test_solution_file_link_tag (tests.test_preup.TestSolutionReplacement) ... ok
test_solution_wrong_link_tag (tests.test_preup.TestSolutionReplacement) ... ok
test_solution_www_bold_tag (tests.test_preup.TestSolutionReplacement) ... ok
test_solution_www_link_tag (tests.test_preup.TestSolutionReplacement) ... ok
test_platform_tag (tests.test_preup.TestXMLUpdates) ... ok
test_result_dirs_current_dir (tests.test_preup.TestXMLUpdates) ... ok
test_result_dirs_tmp_preupgrade (tests.test_preup.TestXMLUpdates) ... ok
test_correct_content_scenario (tests.test_preup.TestScenario) ... ok
test_migration_content_scenario (tests.test_preup.TestScenario) ... ok
test_wrong_content_scenario (tests.test_preup.TestScenario) ... ok
test_correct_prefix (tests.test_preup.TestPreupgradePrefix) ... ok
test_wrong_prefix (tests.test_preup.TestPreupgradePrefix) ... ok
test_correct_prefix (tests.test_preup.TestPremigratePrefix) ... ok
test_wrong_prefix (tests.test_preup.TestPremigratePrefix) ... ok
test_group_ini (tests.test_xml.TestGroupXML)
Basic test creation group.xml file ... ok
test_check_script_applies_to (tests.test_xml.TestXML) ... ok
test_check_script_author (tests.test_xml.TestXML) ... ok
test_check_script_common (tests.test_xml.TestXML) ... ok
test_check_script_requires (tests.test_xml.TestXML) ... ok
test_group_xml (tests.test_xml.TestXML)
Basic test for whole program ... ok
test_values_id (tests.test_xml.TestXML) ... ok
test_xml_check_export_tmp_preupgrade (tests.test_xml.TestXML) ... ok
test_xml_check_script_reference (tests.test_xml.TestXML) ... ok
test_xml_config_file (tests.test_xml.TestXML) ... ok
test_xml_current_directory (tests.test_xml.TestXML) ... ok
test_xml_fix_text (tests.test_xml.TestXML) ... ok
test_xml_migrate_and_upgrade (tests.test_xml.TestXML) ... ok
test_xml_migrate_not_upgrade (tests.test_xml.TestXML) ... ok
test_xml_not_migrate_not_upgrade (tests.test_xml.TestXML) ... ok
test_xml_profile_id (tests.test_xml.TestXML) ... ok
test_xml_rule_id (tests.test_xml.TestXML) ... ok
test_xml_rule_title (tests.test_xml.TestXML) ... ok
test_xml_solution_type_html (tests.test_xml.TestXML) ... ok
test_xml_solution_type_text (tests.test_xml.TestXML) ... ok
test_xml_upgrade_not_migrate (tests.test_xml.TestXML) ... ok
test_check_script_is_directory (tests.test_xml.TestIncorrectINI) ... 

********** ERROR **********
Wrong value for tag 'check_script' in INI file 'tests/FOOBAR6_7/incorrect_ini/test.ini'
 '.': Is a directory


********** ERROR **********
Following tag 'check_script' is missing in INI File tests/FOOBAR6_7/incorrect_ini/test.ini
 
ok
test_file_check_script_not_exists (tests.test_xml.TestIncorrectINI)
Test of missing 'check_script' file ... ERROR:  tests/FOOBAR6_7/incorrect_ini this_should_be_unexpected_file.txt Script name does not exists
List of directory ( tests/FOOBAR6_7/incorrect_ini ) is:
test_solution.sh
check_script.sh
ok
test_file_solution_not_exists (tests.test_xml.TestIncorrectINI)
Test of missing 'solution' file - SystemExit should be raised ... 

********** ERROR **********
The file tests/FOOBAR6_7/incorrect_ini/check_script.sh is not executable 
ERROR:  tests/FOOBAR6_7/incorrect_ini this_should_be_unexpected_file.txt Script name does not exists
List of directory ( tests/FOOBAR6_7/incorrect_ini ) is:
test_solution.sh
check_script.sh
ok
test_incorrect_tag (tests.test_xml.TestIncorrectINI) ... 

********** ERROR **********
Incorrect INI file
 tests/FOOBAR6_7/incorrect_ini/test.ini
ok
test_missing_tag_check_script (tests.test_xml.TestIncorrectINI)
Basic test for whole program ... 

********** ERROR **********
Following tags are missing in INI file tests/FOOBAR6_7/incorrect_ini/test.ini
 check_script
ok
test_missing_tag_solution_script (tests.test_xml.TestIncorrectINI)
Test of missing tag 'solution' - SystemExit should be raised ... 

********** ERROR **********
Following tags are missing in INI file tests/FOOBAR6_7/incorrect_ini/test.ini
 solution
ok
test_secret_check_script (tests.test_xml.TestIncorrectINI)
Check occurrence of secret file for check script ... ERROR:  tests/FOOBAR6_7/incorrect_ini .minicheck Script name does not exists
List of directory ( tests/FOOBAR6_7/incorrect_ini ) is:
test_solution.sh
check_script.sh
ok
test_compose (tests.test_xml.TestXMLCompose)
Basic test of composing ... /usr/lib64/python2.7/unittest/case.py:458: FutureWarning: The behavior of this method will change in future versions.  Use specific 'len(elem)' or 'elem is not None' test instead.
  if not expr:
ok
test_unicode_script_author (tests.test_xml.TestXMLCompose)
Test processing of non-ascii characters for author section ... ok
test_unicode_xml (tests.test_xml.TestXMLCompose) ... ok
test_all (tests.test_xml.HTMLEscapeTest)
Test if list of strings is escaped well ... ok
test_amp_expand (tests.test_xml.HTMLEscapeTest)
Test whether ampersand is not being expanded multiple times ... ok
test_basic (tests.test_xml.HTMLEscapeTest)
Basic test with quotation ... ok
test_basic_string (tests.test_xml.HTMLEscapeTest)
Test if single string is escaped well ... ok
test_pass (tests.test_oscap.TestOSCAPPass) ... creating /tmp/tmpAHoBej/FOOBAR6_7
creating /tmp/tmpAHoBej/FOOBAR6_7/dummy
creating /tmp/tmpAHoBej/FOOBAR6_7/dummy/pass
copying tests/FOOBAR6_7-results/dummy/pass/dummy.ini -> /tmp/tmpAHoBej/FOOBAR6_7/dummy/pass
copying tests/FOOBAR6_7-results/dummy/pass/dummy_pass.sh -> /tmp/tmpAHoBej/FOOBAR6_7/dummy/pass
copying tests/FOOBAR6_7-results/dummy/pass/solution.txt -> /tmp/tmpAHoBej/FOOBAR6_7/dummy/pass
copying tests/FOOBAR6_7-results/dummy/pass/group.xml -> /tmp/tmpAHoBej/FOOBAR6_7/dummy/pass
copying tests/FOOBAR6_7-results/dummy/pass/all-xccdf.xml -> /tmp/tmpAHoBej/FOOBAR6_7/dummy/pass
copying tests/FOOBAR6_7-results/dummy/list_rules -> /tmp/tmpAHoBej/FOOBAR6_7/dummy
copying tests/FOOBAR6_7-results/dummy/migrate -> /tmp/tmpAHoBej/FOOBAR6_7/dummy
copying tests/FOOBAR6_7-results/dummy/upgrade -> /tmp/tmpAHoBej/FOOBAR6_7/dummy
ok
test_fail (tests.test_oscap.TestOSCAPFail) ... creating /tmp/tmp9vWtz8/FOOBAR6_7
creating /tmp/tmp9vWtz8/FOOBAR6_7/dummy
creating /tmp/tmp9vWtz8/FOOBAR6_7/dummy/failed
copying tests/FOOBAR6_7-results/dummy/failed/dummy.ini -> /tmp/tmp9vWtz8/FOOBAR6_7/dummy/failed
copying tests/FOOBAR6_7-results/dummy/failed/solution.txt -> /tmp/tmp9vWtz8/FOOBAR6_7/dummy/failed
copying tests/FOOBAR6_7-results/dummy/failed/dummy_failed.sh -> /tmp/tmp9vWtz8/FOOBAR6_7/dummy/failed
copying tests/FOOBAR6_7-results/dummy/failed/group.xml -> /tmp/tmp9vWtz8/FOOBAR6_7/dummy/failed
copying tests/FOOBAR6_7-results/dummy/failed/all-xccdf.xml -> /tmp/tmp9vWtz8/FOOBAR6_7/dummy/failed
copying tests/FOOBAR6_7-results/dummy/list_rules -> /tmp/tmp9vWtz8/FOOBAR6_7/dummy
copying tests/FOOBAR6_7-results/dummy/migrate -> /tmp/tmp9vWtz8/FOOBAR6_7/dummy
copying tests/FOOBAR6_7-results/dummy/upgrade -> /tmp/tmp9vWtz8/FOOBAR6_7/dummy
ok
test_needs_inspection (tests.test_oscap.TestOSCAPNeedsInspection) ... creating /tmp/tmpNmxK7O/FOOBAR6_7
creating /tmp/tmpNmxK7O/FOOBAR6_7/dummy
creating /tmp/tmpNmxK7O/FOOBAR6_7/dummy/needs_inspection
copying tests/FOOBAR6_7-results/dummy/needs_inspection/dummy.ini -> /tmp/tmpNmxK7O/FOOBAR6_7/dummy/needs_inspection
copying tests/FOOBAR6_7-results/dummy/needs_inspection/dummy_failed.sh -> /tmp/tmpNmxK7O/FOOBAR6_7/dummy/needs_inspection
copying tests/FOOBAR6_7-results/dummy/needs_inspection/solution.txt -> /tmp/tmpNmxK7O/FOOBAR6_7/dummy/needs_inspection
copying tests/FOOBAR6_7-results/dummy/needs_inspection/group.xml -> /tmp/tmpNmxK7O/FOOBAR6_7/dummy/needs_inspection
copying tests/FOOBAR6_7-results/dummy/needs_inspection/all-xccdf.xml -> /tmp/tmpNmxK7O/FOOBAR6_7/dummy/needs_inspection
copying tests/FOOBAR6_7-results/dummy/list_rules -> /tmp/tmpNmxK7O/FOOBAR6_7/dummy
copying tests/FOOBAR6_7-results/dummy/migrate -> /tmp/tmpNmxK7O/FOOBAR6_7/dummy
copying tests/FOOBAR6_7-results/dummy/upgrade -> /tmp/tmpNmxK7O/FOOBAR6_7/dummy
ok
test_needs_action (tests.test_oscap.TestOSCAPNeedsAction) ... creating /tmp/tmpDlZN_q/FOOBAR6_7
creating /tmp/tmpDlZN_q/FOOBAR6_7/dummy
creating /tmp/tmpDlZN_q/FOOBAR6_7/dummy/needs_inspection
copying tests/FOOBAR6_7-results/dummy/needs_inspection/dummy.ini -> /tmp/tmpDlZN_q/FOOBAR6_7/dummy/needs_inspection
copying tests/FOOBAR6_7-results/dummy/needs_inspection/dummy_failed.sh -> /tmp/tmpDlZN_q/FOOBAR6_7/dummy/needs_inspection
copying tests/FOOBAR6_7-results/dummy/needs_inspection/solution.txt -> /tmp/tmpDlZN_q/FOOBAR6_7/dummy/needs_inspection
copying tests/FOOBAR6_7-results/dummy/needs_inspection/group.xml -> /tmp/tmpDlZN_q/FOOBAR6_7/dummy/needs_inspection
copying tests/FOOBAR6_7-results/dummy/needs_inspection/all-xccdf.xml -> /tmp/tmpDlZN_q/FOOBAR6_7/dummy/needs_inspection
copying tests/FOOBAR6_7-results/dummy/migrate -> /tmp/tmpDlZN_q/FOOBAR6_7/dummy
copying tests/FOOBAR6_7-results/dummy/upgrade -> /tmp/tmpDlZN_q/FOOBAR6_7/dummy
creating /tmp/tmpDlZN_q/FOOBAR6_7/dummy/needs_action
copying tests/FOOBAR6_7-results/dummy/needs_action/dummy.ini -> /tmp/tmpDlZN_q/FOOBAR6_7/dummy/needs_action
copying tests/FOOBAR6_7-results/dummy/needs_action/dummy_failed.sh -> /tmp/tmpDlZN_q/FOOBAR6_7/dummy/needs_action
copying tests/FOOBAR6_7-results/dummy/needs_action/solution.txt -> /tmp/tmpDlZN_q/FOOBAR6_7/dummy/needs_action
copying tests/FOOBAR6_7-results/dummy/needs_action/group.xml -> /tmp/tmpDlZN_q/FOOBAR6_7/dummy/needs_action
copying tests/FOOBAR6_7-results/dummy/needs_action/all-xccdf.xml -> /tmp/tmpDlZN_q/FOOBAR6_7/dummy/needs_action
copying tests/FOOBAR6_7-results/dummy/list_rules -> /tmp/tmpDlZN_q/FOOBAR6_7/dummy
ok
test_not_applicable (tests.test_oscap.TestOSCAPNotApplicable) ... creating /tmp/tmpemFZqJ/FOOBAR6_7
creating /tmp/tmpemFZqJ/FOOBAR6_7/dummy
creating /tmp/tmpemFZqJ/FOOBAR6_7/dummy/not_applicable
copying tests/FOOBAR6_7-results/dummy/not_applicable/dummy.ini -> /tmp/tmpemFZqJ/FOOBAR6_7/dummy/not_applicable
copying tests/FOOBAR6_7-results/dummy/not_applicable/dummy_not_applicable.sh -> /tmp/tmpemFZqJ/FOOBAR6_7/dummy/not_applicable
copying tests/FOOBAR6_7-results/dummy/not_applicable/solution.txt -> /tmp/tmpemFZqJ/FOOBAR6_7/dummy/not_applicable
copying tests/FOOBAR6_7-results/dummy/not_applicable/group.xml -> /tmp/tmpemFZqJ/FOOBAR6_7/dummy/not_applicable
copying tests/FOOBAR6_7-results/dummy/not_applicable/all-xccdf.xml -> /tmp/tmpemFZqJ/FOOBAR6_7/dummy/not_applicable
copying tests/FOOBAR6_7-results/dummy/list_rules -> /tmp/tmpemFZqJ/FOOBAR6_7/dummy
copying tests/FOOBAR6_7-results/dummy/migrate -> /tmp/tmpemFZqJ/FOOBAR6_7/dummy
copying tests/FOOBAR6_7-results/dummy/upgrade -> /tmp/tmpemFZqJ/FOOBAR6_7/dummy
ok
test_fixed (tests.test_oscap.TestOSCAPFixed) ... creating /tmp/tmpMWECdk/FOOBAR6_7
creating /tmp/tmpMWECdk/FOOBAR6_7/dummy
creating /tmp/tmpMWECdk/FOOBAR6_7/dummy/fixed
copying tests/FOOBAR6_7-results/dummy/fixed/dummy.ini -> /tmp/tmpMWECdk/FOOBAR6_7/dummy/fixed
copying tests/FOOBAR6_7-results/dummy/fixed/dummy_fixed.sh -> /tmp/tmpMWECdk/FOOBAR6_7/dummy/fixed
copying tests/FOOBAR6_7-results/dummy/fixed/solution.txt -> /tmp/tmpMWECdk/FOOBAR6_7/dummy/fixed
copying tests/FOOBAR6_7-results/dummy/fixed/group.xml -> /tmp/tmpMWECdk/FOOBAR6_7/dummy/fixed
copying tests/FOOBAR6_7-results/dummy/fixed/all-xccdf.xml -> /tmp/tmpMWECdk/FOOBAR6_7/dummy/fixed
copying tests/FOOBAR6_7-results/dummy/list_rules -> /tmp/tmpMWECdk/FOOBAR6_7/dummy
copying tests/FOOBAR6_7-results/dummy/migrate -> /tmp/tmpMWECdk/FOOBAR6_7/dummy
copying tests/FOOBAR6_7-results/dummy/upgrade -> /tmp/tmpMWECdk/FOOBAR6_7/dummy
ok
test_check_inplace_risk_extreme (tests.test_api.TestRiskCheck) ... ok
test_check_inplace_risk_high (tests.test_api.TestRiskCheck) ... ok
test_check_inplace_risk_medium (tests.test_api.TestRiskCheck) ... ok
test_check_inplace_risk_none (tests.test_api.TestRiskCheck) ... ok
test_check_inplace_risk_slight (tests.test_api.TestRiskCheck) ... ok
test_check_inplace_risk_unknown (tests.test_api.TestRiskCheck) ... ok
test_crypt_partitions (tests.test_kickstart.TestPartitioning) ... ok
test_lvm_crypt_partitions (tests.test_kickstart.TestPartitioning) ... ok
test_lvm_partitions (tests.test_kickstart.TestPartitioning) ... ok
test_native_partitioning (tests.test_kickstart.TestPartitioning) ... ok
test_raid_crypt_partitions (tests.test_kickstart.TestPartitioning) ... ok
test_raid_second_partitions (tests.test_kickstart.TestPartitioning) ... ok

----------------------------------------------------------------------
Ran 74 tests in 17.892s

OK
Note You need to log in before you can comment on or make changes to this bug.