Bug 2166612

Summary: python-TestSlide FTBFS with Pygments 2.14
Product: [Fedora] Fedora Reporter: Karolina Surma <ksurma>
Component: python-TestSlideAssignee: Davide Cavalca <davide>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 38CC: code, davide, filbranden, michel
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-TestSlide-2.7.1-1.fc39 python-TestSlide-2.7.1-2.fc38 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-06-26 16:51:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2179100    
Bug Blocks: 2157264    

Description Karolina Surma 2023-02-02 10:35:49 UTC 
Description of problem:
python-TestSlide fails to build with Pygments 2.14 we want to bring to Rawhide. 
It's likely related to some output changes brought by new Pygments. I checked the impact in a copr repository, see links below.

Version-Release number of selected component (if applicable):
2.7.0-7

How reproducible:
Always

Steps to Reproduce:
$ mock -r fedora-rawhide-x86_64 --addrepo=https://download.copr.fedorainfracloud.org/results/ksurma/pygments-2.14/fedora-rawhide-x86_64/ --no-clean your.src.rpm
$ mock -r fedora-rawhide-x86_64 --addrepo=https://download.copr.fedorainfracloud.org/results/ksurma/pygments-2.14/fedora-rawhide-x86_64/  shell

Actual results:
https://download.copr.fedorainfracloud.org/results/ksurma/pygments-2.14/fedora-rawhide-x86_64/05362858-python-TestSlide/builder-live.log.gz


=================================== FAILURES ===================================
__________ TestCliDocumentFormatter.test_prints_exceptions_with_cause __________

self = <tests.cli_unittest.TestCliDocumentFormatter testMethod=test_prints_exceptions_with_cause>

    def test_prints_exceptions_with_cause(self):
>       self.run_testslide(
            tty_stdout=True,
            expected_return_code=1,
            expected_in_stdout=(
                '      File \033[36m"tests/sample_tests.py"\033[39;49;00m, line \033[34m76\033[39;49;00m, in test_failing\r\n'
                '        \033[34mraise\033[39;49;00m \033[36mAssertionError\033[39;49;00m(\033[33m"\033[39;49;00m\033[33mThird\033[39;49;00m\033[33m"\033[39;49;00m) \033[34mfrom\033[39;49;00m \033[04m\033[36mcause\033[39;49;00m\r\n'
                "\033[0m\033[31m      Caused by \033[0m\033[0m\033[31mAssertionError: Second\033[0m\r\n"
                '        File \033[36m"tests/sample_tests.py"\033[39;49;00m, line \033[34m74\033[39;49;00m, in test_failing\r\n'
                '          \033[34mraise\033[39;49;00m \033[36mAssertionError\033[39;49;00m(\033[33m"\033[39;49;00m\033[33mSecond\033[39;49;00m\033[33m"\033[39;49;00m) \033[34mfrom\033[39;49;00m \033[04m\033[36mcause\033[39;49;00m\r\n'
                "\033[0m\033[31m        Caused by \033[0m\033[0m\033[31mAssertionError: First\033[0m\r\n"
                '          File \033[36m"tests/sample_tests.py"\033[39;49;00m, line \033[34m72\033[39;49;00m, in test_failing\r\n'
                '            \033[34mraise\033[39;49;00m \033[36mAssertionError\033[39;49;00m(\033[33m"\033[39;49;00m\033[33mFirst\033[39;49;00m\033[33m"\033[39;49;00m)\r\n'
            ),
        )

tests/cli_unittest.py:284: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/cli_unittest.py:137: in run_testslide
    self.assertTrue(
E   AssertionError: False is not true : Command ['/usr/bin/python3', '-m', 'testslide.cli', '--show-testslide-stack-trace', '--format', 'documentation', '/builddir/build/BUILD/TestSlide-2.7.0/tests/sample_tests.py'] expected to have have in its stdout:
E   
E         File "tests/sample_tests.py", line 76, in test_failing
E           raise AssertionError("Third") from cause
E         Caused by AssertionError: Second
E           File "tests/sample_tests.py", line 74, in test_failing
E             raise AssertionError("Second") from cause
E           Caused by AssertionError: First
E             File "tests/sample_tests.py", line 72, in test_failing
E               raise AssertionError("First")
E   
E   
E   But output was different:
E   top context
E     passing example
E     failing example: SimulatedFailure: test failure (extra)
E     *focused example
E     skipped example
E     unittest SkipTest
E     nested context
E       passing nested example
E   tests.sample_tests.SampleTestCase
E     test_failing: AssertionError: Third
E     test_passing
E     test_skipped
E   
E   Failures:
E   
E     1) top context: failing example
E       1) SimulatedFailure: test failure (extra)
E         File "testslide/runner.py", line 789, in run
E           self._run_example(example)
E         File "testslide/runner.py", line 773, in _run_example
E           _ExampleRunner(example, self.formatter).run()
E         File "testslide/__init__.py", line 540, in run
E           self._sync_run_all_hooks_and_example(context_data)
E         File "testslide/__init__.py", line 512, in _sync_run_all_hooks_and_example
E           aggregated_exceptions.raise_correct_exception()
E         File "testslide/__init__.py", line 289, in raise_correct_exception
E           raise self.exceptions[0]
E         File "testslide/__init__.py", line 273, in catch
E           yield
E         File "testslide/__init__.py", line 503, in _sync_run_all_hooks_and_example
E           self._fail_if_coroutine_function(self.example.code, context_data)
E         File "testslide/__init__.py", line 471, in _fail_if_coroutine_function
E           return func(*args, **kwargs)
E         File "tests/sample_tests.py", line 42, in failing_example
E           raise SimulatedFailure("test failure", "(extra)")
E   
E     2) tests.sample_tests.SampleTestCase: test_failing
E       1) AssertionError: Third
E         File "testslide/runner.py", line 789, in run
E           self._run_example(example)
E         File "testslide/runner.py", line 773, in _run_example
E           _ExampleRunner(example, self.formatter).run()
E         File "testslide/__init__.py", line 540, in run
E           self._sync_run_all_hooks_and_example(context_data)
E         File "testslide/__init__.py", line 512, in _sync_run_all_hooks_and_example
E           aggregated_exceptions.raise_correct_exception()
E         File "testslide/__init__.py", line 289, in raise_correct_exception
E           raise self.exceptions[0]
E         File "testslide/__init__.py", line 273, in catch
E           yield
E         File "testslide/__init__.py", line 503, in _sync_run_all_hooks_and_example
E           self._fail_if_coroutine_function(self.example.code, context_data)
E         File "testslide/__init__.py", line 471, in _fail_if_coroutine_function
E           return func(*args, **kwargs)
E         File "testslide/cli.py", line 121, in example_code
E           with test_result() as result:
E         File "/usr/lib64/python3.11/contextlib.py", line 144, in __exit__
E           next(self.gen)
E         File "testslide/cli.py", line 110, in test_result
E           result.aggregated_exceptions.raise_correct_exception()
E         File "testslide/__init__.py", line 289, in raise_correct_exception
E           raise self.exceptions[0]
E         File "/usr/lib64/python3.11/unittest/case.py", line 57, in testPartExecutor
E           yield
E         File "/usr/lib64/python3.11/unittest/case.py", line 623, in run
E           self._callTestMethod(testMethod)
E         File "/usr/lib64/python3.11/unittest/case.py", line 579, in _callTestMethod
E           if method() is not None:
E         File "tests/sample_tests.py", line 76, in test_failing
E           raise AssertionError("Third") from cause
E         Caused by AssertionError: Second
E           File "tests/sample_tests.py", line 74, in test_failing
E             raise AssertionError("Second") from cause
E           Caused by AssertionError: First
E             File "tests/sample_tests.py", line 72, in test_failing
E               raise AssertionError("First")
E   
E   Executed 9 examples in 0.3s:
E     Successful: 4
E     Failed: 2
E     Skipped: 3
E     Not executed: 0
E   https://testslide.readthedocs.io/
__________ TestCliProgressFormatter.test_prints_exceptions_with_cause __________

self = <tests.cli_unittest.TestCliProgressFormatter testMethod=test_prints_exceptions_with_cause>

    def test_prints_exceptions_with_cause(self):
>       self.run_testslide(
            tty_stdout=True,
            expected_return_code=1,
            expected_in_stdout=(
                '      File \033[36m"tests/sample_tests.py"\033[39;49;00m, line \033[34m76\033[39;49;00m, in test_failing\r\n'
                '        \033[34mraise\033[39;49;00m \033[36mAssertionError\033[39;49;00m(\033[33m"\033[39;49;00m\033[33mThird\033[39;49;00m\033[33m"\033[39;49;00m) \033[34mfrom\033[39;49;00m \033[04m\033[36mcause\033[39;49;00m\r\n'
                "\033[0m\033[31m      Caused by \033[0m\033[0m\033[31mAssertionError: Second\033[0m\r\n"
                '        File \033[36m"tests/sample_tests.py"\033[39;49;00m, line \033[34m74\033[39;49;00m, in test_failing\r\n'
                '          \033[34mraise\033[39;49;00m \033[36mAssertionError\033[39;49;00m(\033[33m"\033[39;49;00m\033[33mSecond\033[39;49;00m\033[33m"\033[39;49;00m) \033[34mfrom\033[39;49;00m \033[04m\033[36mcause\033[39;49;00m\r\n'
                "\033[0m\033[31m        Caused by \033[0m\033[0m\033[31mAssertionError: First\033[0m\r\n"
                '          File \033[36m"tests/sample_tests.py"\033[39;49;00m, line \033[34m72\033[39;49;00m, in test_failing\r\n'
                '            \033[34mraise\033[39;49;00m \033[36mAssertionError\033[39;49;00m(\033[33m"\033[39;49;00m\033[33mFirst\033[39;49;00m\033[33m"\033[39;49;00m)\r\n'
            ),
        )

tests/cli_unittest.py:284: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/cli_unittest.py:137: in run_testslide
    self.assertTrue(
E   AssertionError: False is not true : Command ['/usr/bin/python3', '-m', 'testslide.cli', '--show-testslide-stack-trace', '/builddir/build/BUILD/TestSlide-2.7.0/tests/sample_tests.py', '--format', 'progress'] expected to have have in its stdout:
E   
E         File "tests/sample_tests.py", line 76, in test_failing
E           raise AssertionError("Third") from cause
E         Caused by AssertionError: Second
E           File "tests/sample_tests.py", line 74, in test_failing
E             raise AssertionError("Second") from cause
E           Caused by AssertionError: First
E             File "tests/sample_tests.py", line 72, in test_failing
E               raise AssertionError("First")
E   
E   
E   But output was different:
E   .F.SS.F.S
E   Failures:
E   
E     1) top context: failing example
E       1) SimulatedFailure: test failure (extra)
E         File "testslide/runner.py", line 789, in run
E           self._run_example(example)
E         File "testslide/runner.py", line 773, in _run_example
E           _ExampleRunner(example, self.formatter).run()
E         File "testslide/__init__.py", line 540, in run
E           self._sync_run_all_hooks_and_example(context_data)
E         File "testslide/__init__.py", line 512, in _sync_run_all_hooks_and_example
E           aggregated_exceptions.raise_correct_exception()
E         File "testslide/__init__.py", line 289, in raise_correct_exception
E           raise self.exceptions[0]
E         File "testslide/__init__.py", line 273, in catch
E           yield
E         File "testslide/__init__.py", line 503, in _sync_run_all_hooks_and_example
E           self._fail_if_coroutine_function(self.example.code, context_data)
E         File "testslide/__init__.py", line 471, in _fail_if_coroutine_function
E           return func(*args, **kwargs)
E         File "tests/sample_tests.py", line 42, in failing_example
E           raise SimulatedFailure("test failure", "(extra)")
E   
E     2) tests.sample_tests.SampleTestCase: test_failing
E       1) AssertionError: Third
E         File "testslide/runner.py", line 789, in run
E           self._run_example(example)
E         File "testslide/runner.py", line 773, in _run_example
E           _ExampleRunner(example, self.formatter).run()
E         File "testslide/__init__.py", line 540, in run
E           self._sync_run_all_hooks_and_example(context_data)
E         File "testslide/__init__.py", line 512, in _sync_run_all_hooks_and_example
E           aggregated_exceptions.raise_correct_exception()
E         File "testslide/__init__.py", line 289, in raise_correct_exception
E           raise self.exceptions[0]
E         File "testslide/__init__.py", line 273, in catch
E           yield
E         File "testslide/__init__.py", line 503, in _sync_run_all_hooks_and_example
E           self._fail_if_coroutine_function(self.example.code, context_data)
E         File "testslide/__init__.py", line 471, in _fail_if_coroutine_function
E           return func(*args, **kwargs)
E         File "testslide/cli.py", line 121, in example_code
E           with test_result() as result:
E         File "/usr/lib64/python3.11/contextlib.py", line 144, in __exit__
E           next(self.gen)
E         File "testslide/cli.py", line 110, in test_result
E           result.aggregated_exceptions.raise_correct_exception()
E         File "testslide/__init__.py", line 289, in raise_correct_exception
E           raise self.exceptions[0]
E         File "/usr/lib64/python3.11/unittest/case.py", line 57, in testPartExecutor
E           yield
E         File "/usr/lib64/python3.11/unittest/case.py", line 623, in run
E           self._callTestMethod(testMethod)
E         File "/usr/lib64/python3.11/unittest/case.py", line 579, in _callTestMethod
E           if method() is not None:
E         File "tests/sample_tests.py", line 76, in test_failing
E           raise AssertionError("Third") from cause
E         Caused by AssertionError: Second
E           File "tests/sample_tests.py", line 74, in test_failing
E             raise AssertionError("Second") from cause
E           Caused by AssertionError: First
E             File "tests/sample_tests.py", line 72, in test_failing
E               raise AssertionError("First")
____________ TestCliLongFormatter.test_prints_exceptions_with_cause ____________

self = <tests.cli_unittest.TestCliLongFormatter testMethod=test_prints_exceptions_with_cause>

    def test_prints_exceptions_with_cause(self):
>       self.run_testslide(
            tty_stdout=True,
            expected_return_code=1,
            expected_in_stdout=(
                '      File \033[36m"tests/sample_tests.py"\033[39;49;00m, line \033[34m76\033[39;49;00m, in test_failing\r\n'
                '        \033[34mraise\033[39;49;00m \033[36mAssertionError\033[39;49;00m(\033[33m"\033[39;49;00m\033[33mThird\033[39;49;00m\033[33m"\033[39;49;00m) \033[34mfrom\033[39;49;00m \033[04m\033[36mcause\033[39;49;00m\r\n'
                "\033[0m\033[31m      Caused by \033[0m\033[0m\033[31mAssertionError: Second\033[0m\r\n"
                '        File \033[36m"tests/sample_tests.py"\033[39;49;00m, line \033[34m74\033[39;49;00m, in test_failing\r\n'
                '          \033[34mraise\033[39;49;00m \033[36mAssertionError\033[39;49;00m(\033[33m"\033[39;49;00m\033[33mSecond\033[39;49;00m\033[33m"\033[39;49;00m) \033[34mfrom\033[39;49;00m \033[04m\033[36mcause\033[39;49;00m\r\n'
                "\033[0m\033[31m        Caused by \033[0m\033[0m\033[31mAssertionError: First\033[0m\r\n"
                '          File \033[36m"tests/sample_tests.py"\033[39;49;00m, line \033[34m72\033[39;49;00m, in test_failing\r\n'
                '            \033[34mraise\033[39;49;00m \033[36mAssertionError\033[39;49;00m(\033[33m"\033[39;49;00m\033[33mFirst\033[39;49;00m\033[33m"\033[39;49;00m)\r\n'
            ),
        )

tests/cli_unittest.py:284: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/cli_unittest.py:137: in run_testslide
    self.assertTrue(
E   AssertionError: False is not true : Command ['/usr/bin/python3', '-m', 'testslide.cli', '--show-testslide-stack-trace', '--format', 'long', '/builddir/build/BUILD/TestSlide-2.7.0/tests/sample_tests.py'] expected to have have in its stdout:
E   
E         File "tests/sample_tests.py", line 76, in test_failing
E           raise AssertionError("Third") from cause
E         Caused by AssertionError: Second
E           File "tests/sample_tests.py", line 74, in test_failing
E             raise AssertionError("Second") from cause
E           Caused by AssertionError: First
E             File "tests/sample_tests.py", line 72, in test_failing
E               raise AssertionError("First")
E   
E   
E   But output was different:
E   top context: passing example
E   top context: failing example: SimulatedFailure: test failure (extra)
E   top context: *focused example
E   top context: skipped example
E   top context: unittest SkipTest
E   top context, nested context: passing nested example
E   tests.sample_tests.SampleTestCase: test_failing: AssertionError: Third
E   tests.sample_tests.SampleTestCase: test_passing
E   tests.sample_tests.SampleTestCase: test_skipped
E   
E   Failures:
E   
E     1) top context: failing example
E       1) SimulatedFailure: test failure (extra)
E         File "testslide/runner.py", line 789, in run
E           self._run_example(example)
E         File "testslide/runner.py", line 773, in _run_example
E           _ExampleRunner(example, self.formatter).run()
E         File "testslide/__init__.py", line 540, in run
E           self._sync_run_all_hooks_and_example(context_data)
E         File "testslide/__init__.py", line 512, in _sync_run_all_hooks_and_example
E           aggregated_exceptions.raise_correct_exception()
E         File "testslide/__init__.py", line 289, in raise_correct_exception
E           raise self.exceptions[0]
E         File "testslide/__init__.py", line 273, in catch
E           yield
E         File "testslide/__init__.py", line 503, in _sync_run_all_hooks_and_example
E           self._fail_if_coroutine_function(self.example.code, context_data)
E         File "testslide/__init__.py", line 471, in _fail_if_coroutine_function
E           return func(*args, **kwargs)
E         File "tests/sample_tests.py", line 42, in failing_example
E           raise SimulatedFailure("test failure", "(extra)")
E   
E     2) tests.sample_tests.SampleTestCase: test_failing
E       1) AssertionError: Third
E         File "testslide/runner.py", line 789, in run
E           self._run_example(example)
E         File "testslide/runner.py", line 773, in _run_example
E           _ExampleRunner(example, self.formatter).run()
E         File "testslide/__init__.py", line 540, in run
E           self._sync_run_all_hooks_and_example(context_data)
E         File "testslide/__init__.py", line 512, in _sync_run_all_hooks_and_example
E           aggregated_exceptions.raise_correct_exception()
E         File "testslide/__init__.py", line 289, in raise_correct_exception
E           raise self.exceptions[0]
E         File "testslide/__init__.py", line 273, in catch
E           yield
E         File "testslide/__init__.py", line 503, in _sync_run_all_hooks_and_example
E           self._fail_if_coroutine_function(self.example.code, context_data)
E         File "testslide/__init__.py", line 471, in _fail_if_coroutine_function
E           return func(*args, **kwargs)
E         File "testslide/cli.py", line 121, in example_code
E           with test_result() as result:
E         File "/usr/lib64/python3.11/contextlib.py", line 144, in __exit__
E           next(self.gen)
E         File "testslide/cli.py", line 110, in test_result
E           result.aggregated_exceptions.raise_correct_exception()
E         File "testslide/__init__.py", line 289, in raise_correct_exception
E           raise self.exceptions[0]
E         File "/usr/lib64/python3.11/unittest/case.py", line 57, in testPartExecutor
E           yield
E         File "/usr/lib64/python3.11/unittest/case.py", line 623, in run
E           self._callTestMethod(testMethod)
E         File "/usr/lib64/python3.11/unittest/case.py", line 579, in _callTestMethod
E           if method() is not None:
E         File "tests/sample_tests.py", line 76, in test_failing
E           raise AssertionError("Third") from cause
E         Caused by AssertionError: Second
E           File "tests/sample_tests.py", line 74, in test_failing
E             raise AssertionError("Second") from cause
E           Caused by AssertionError: First
E             File "tests/sample_tests.py", line 72, in test_failing
E               raise AssertionError("First")
E   
E   Executed 9 examples in 0.3s:
E     Successful: 4
E     Failed: 2
E     Skipped: 3
E     Not executed: 0
E   https://testslide.readthedocs.io/
=========================== short test summary info ============================
FAILED tests/cli_unittest.py::TestCliDocumentFormatter::test_prints_exceptions_with_cause
FAILED tests/cli_unittest.py::TestCliProgressFormatter::test_prints_exceptions_with_cause
FAILED tests/cli_unittest.py::TestCliLongFormatter::test_prints_exceptions_with_cause

Expected results:
Success
Comment 1 Davide Cavalca 2023-02-03 12:28:03 UTC 
Reported upstream in https://github.com/facebook/TestSlide/issues/345
Comment 2 Ben Cotton 2023-02-07 15:12:24 UTC 
This bug appears to have been reported against 'rawhide' during the Fedora Linux 38 development cycle.
Changing version to 38.
Comment 3 Ben Beasley 2023-05-12 14:16:13 UTC 
https://src.fedoraproject.org/rpms/python-TestSlide/pull-request/1
Comment 4 Fedora Update System 2023-06-26 16:48:11 UTC 
FEDORA-2023-a2d687e102 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-a2d687e102
Comment 5 Fedora Update System 2023-06-26 16:51:07 UTC 
FEDORA-2023-a2d687e102 has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.
Comment 6 Fedora Update System 2023-06-26 17:40:52 UTC 
FEDORA-2023-becc1c0336 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-becc1c0336
Comment 7 Fedora Update System 2023-06-27 10:55:39 UTC 
FEDORA-2023-becc1c0336 has been pushed to the Fedora 38 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-becc1c0336`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-becc1c0336

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Comment 8 Fedora Update System 2023-06-29 01:46:14 UTC 
FEDORA-2023-becc1c0336 has been pushed to the Fedora 38 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-becc1c0336`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-becc1c0336

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Comment 9 Fedora Update System 2023-07-07 01:28:33 UTC 
FEDORA-2023-becc1c0336 has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.