Bug 2102736 - invoke fails under python 3.11 with AttributeError
Summary: invoke fails under python 3.11 with AttributeError
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: python-invoke
Version: 37
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Jiri Kucera
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: PYTHON3.11 2098908 2113626
TreeView+ depends on / blocked
 
Reported: 2022-06-30 13:48 UTC by Dominik 'Rathann' Mierzejewski
Modified: 2023-02-14 01:52 UTC (History)
3 users (show)

Fixed In Version: python-invoke-1.7.0-5.fc38 python-invoke-1.7.0-6.fc37
Clone Of:
Environment:
Last Closed: 2023-02-14 01:52:15 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Dominik 'Rathann' Mierzejewski 2022-06-30 13:48:30 UTC
Description of problem:
invoke fails with "AttributeError: module 'inspect' has no attribute 'getargspec'." under python 3.11 when trying to run python-filecheck test suite.

Version-Release number of selected component (if applicable):
python3-invoke-1.7.0-2.fc37.noarch

How reproducible:
Always.

Steps to Reproduce:
1. wget https://kojipkgs.fedoraproject.org//packages/python-filecheck/0.0.18/2.fc36/src/python-filecheck-0.0.18-2.fc36.src.rpm
2. nice mock -r fedora-rawhide-x86_64 python-filecheck-0.0.18-2.fc36.src.rpm

Actual results:
...
+ /usr/bin/invoke -e test
Traceback (most recent call last):
  File "/usr/bin/invoke", line 33, in <module>
    sys.exit(load_entry_point('invoke==1.7.0', 'console_scripts', 'invoke')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/invoke/program.py", line 373, in run
    self.parse_collection()
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/invoke/program.py", line 465, in parse_collection
    self.load_collection()
    ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/invoke/program.py", line 696, in load_collection
    module, parent = loader.load(coll_name)
                     ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/invoke/loader.py", line 76, in load
    module = imp.load_module(name, fd, path, desc)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/imp.py", line 235, in load_module
    return load_source(name, filename, file)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/imp.py", line 172, in load_source
    module = _load(spec)
             ^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 721, in _load
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 939, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/builddir/build/BUILD/FileCheck.py-0.0.22/tasks.py", line 77, in <module>
    @task
     ^^^^
  File "/usr/lib/python3.11/site-packages/invoke/tasks.py", line 331, in task
    return klass(args[0], **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/invoke/tasks.py", line 76, in __init__
    self.positional = self.fill_implicit_positionals(positional)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/invoke/tasks.py", line 167, in fill_implicit_positionals
    args, spec_dict = self.argspec(self.body)
                      ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/invoke/tasks.py", line 153, in argspec
    spec = inspect.getargspec(func)
           ^^^^^^^^^^^^^^^^^^
AttributeError: module 'inspect' has no attribute 'getargspec'. Did you mean: 'getargs'?
error: Bad exit status from /var/tmp/rpm-tmp.p7T7os (%check)

RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.p7T7os (%check)

Expected results:
Successful execution.

Comment 1 Ben Cotton 2022-08-09 13:19:43 UTC
This bug appears to have been reported against 'rawhide' during the Fedora Linux 37 development cycle.
Changing version to 37.

Comment 2 Jiri Kucera 2022-09-02 06:53:15 UTC
s/inspect.getargspec/inspect.getfullargspec/g + necessary code tweaks should fix this. Since %check is disabled due to orphaned pytest-relaxed, I am afraid there can be more errors like this. I'll try to fix and unorphan pytest-relaxed first and then fix this and all remaining flaws.

Comment 4 Fedora Update System 2022-09-15 09:06:31 UTC
FEDORA-2022-50b50923b0 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2022-50b50923b0

Comment 5 Fedora Update System 2022-09-15 09:10:06 UTC
FEDORA-2022-50b50923b0 has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 6 Dominik 'Rathann' Mierzejewski 2022-09-15 09:21:00 UTC
Sorry, didn't mean to close this with my update.

Comment 7 Miro Hrončok 2022-09-15 12:59:21 UTC
Has there been some progress here?

Comment 8 Jiri Kucera 2022-09-16 21:08:41 UTC
Thanks for the fix in Comment 3, Miro. I filed request for python-pytest-relaxed unretirement: bz#2127549

Comment 9 Miro Hrončok 2022-09-28 22:39:24 UTC
With tests and s/getargspec/getfullargspec/:

+ pytest-3
============================= test session starts ==============================
platform linux -- Python 3.11.0rc2, pytest-7.1.3, pluggy-1.0.0
rootdir: /builddir/build/BUILD/invoke-1.7.0, configfile: pytest.ini, testpaths: tests
plugins: relaxed-1.1.5
collected 980 items

tests/cli.py ......................                                      [  2%]
tests/collection.py .................................................... [  7%]
......................................                                   [ 11%]
tests/completion.py ........................                             [ 13%]
tests/concurrency.py ......                                              [ 14%]
tests/config.py ........................................................ [ 20%]
.............................................................            [ 26%]
tests/context.py .................................s.s....FF.F........... [ 32%]
.............................                                            [ 35%]
tests/executor.py .....................................                  [ 38%]
tests/init.py ............................                               [ 41%]
tests/loader.py ...............                                          [ 43%]
tests/merge_dicts.py ............                                        [ 44%]
tests/parser_argument.py ..................s...................          [ 48%]
tests/parser_context.py .............................................    [ 52%]
tests/parser_parser.py ................................................. [ 57%]
...........                                                              [ 58%]
tests/program.py ....................................................... [ 64%]
...ss..............................................................      [ 71%]
tests/runners.py ....sssssssssssssssssssssssssssssssssssF..sss..ssssssss [ 77%]
..ssssssssssss......ss..sssssssssssssssssssssssssss.........s.sssss.ssss [ 84%]
s............ssssss.ssssss.ss.s..s..........................             [ 90%]
tests/task.py ......s................................................... [ 96%]
.............                                                            [ 97%]
tests/terminals.py ......ss                                              [ 98%]
tests/util.py .......                                                    [ 99%]
tests/watchers.py .......                                                [100%]

=================================== FAILURES ===================================
_ Context_.sudo.auto_response_merges_with_other_responses.kwarg_only_adds_to_kwarg _

self = <context.Context_.sudo.auto_response_merges_with_other_responses object at 0x7f232f5fbb90>
Local = <MagicMock name='Local' id='139789090720208'>

    @patch(local_path)
    def kwarg_only_adds_to_kwarg(self, Local):
        runner = Local.return_value
        context = Context()
        watcher = self.watcher_klass()
        context.sudo("whoami", watchers=[watcher])
        # When sudo() called w/ user-specified watchers, we add ours to
        # that list
        watchers = runner.run.call_args[1]["watchers"]
        # Will raise ValueError if not in the list
        watchers.remove(watcher)
        # Only remaining item in list should be our sudo responder
        assert len(watchers) == 1
        assert isinstance(watchers[0], FailingResponder)
>       assert watchers[0].pattern == self.escaped_prompt
E       AttributeError: 'auto_response_merges_with_other_responses' object has no attribute 'escaped_prompt'

tests/context.py:431: AttributeError
_____ Context_.sudo.auto_response_merges_with_other_responses.config_only ______

self = <context.Context_.sudo.auto_response_merges_with_other_responses object at 0x7f232f64db10>
Local = <MagicMock name='Local' id='139789093869200'>

    @patch(local_path)
    def config_only(self, Local):
        runner = Local.return_value
        # Set a config-driven list of watchers
        watcher = self.watcher_klass()
        overrides = {"run": {"watchers": [watcher]}}
        config = Config(overrides=overrides)
        Context(config=config).sudo("whoami")
        # Expect that sudo() extracted that config value & put it into
        # the kwarg level. (See comment in sudo() about why...)
        watchers = runner.run.call_args[1]["watchers"]
        # Will raise ValueError if not in the list
        watchers.remove(watcher)
        # Only remaining item in list should be our sudo responder
        assert len(watchers) == 1
        assert isinstance(watchers[0], FailingResponder)
>       assert watchers[0].pattern == self.escaped_prompt
E       AttributeError: 'auto_response_merges_with_other_responses' object has no attribute 'escaped_prompt'

tests/context.py:449: AttributeError
_ Context_.sudo.auto_response_merges_with_other_responses.both_kwarg_and_config _

self = <context.Context_.sudo.auto_response_merges_with_other_responses object at 0x7f232f64d990>
Local = <MagicMock name='Local' id='139789086725904'>

    @patch(local_path)
    def both_kwarg_and_config(self, Local):
        runner = Local.return_value
        # Set a config-driven list of watchers
        conf_watcher = self.watcher_klass()
        overrides = {"run": {"watchers": [conf_watcher]}}
        config = Config(overrides=overrides)
        # AND supply a DIFFERENT kwarg-driven list of watchers
        kwarg_watcher = self.watcher_klass()
        Context(config=config).sudo("whoami", watchers=[kwarg_watcher])
        # Expect that the kwarg watcher and our internal one were the
        # final result.
        watchers = runner.run.call_args[1]["watchers"]
        # Will raise ValueError if not in the list. .remove() uses
        # identity testing, so two instances of self.watcher_klass will
        # be different values here.
        watchers.remove(kwarg_watcher)
        # Only remaining item in list should be our sudo responder
        assert len(watchers) == 1
        assert conf_watcher not in watchers  # Extra sanity
        assert isinstance(watchers[0], FailingResponder)
>       assert watchers[0].pattern == self.escaped_prompt
E       AttributeError: 'auto_response_merges_with_other_responses' object has no attribute 'escaped_prompt'

tests/context.py:493: AttributeError
__________________ Runner_.command_echoing.uses_custom_format __________________

self = <runners.Runner_.command_echoing object at 0x7f232ed30990>

    @trap
    def uses_custom_format(self):
>       self._run(
            "my command",
            echo=True,
            settings={"run": {"echo_format": "AA{command}ZZ"}},
        )

tests/runners.py:385: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/runners.py:138: in _run
    return _run(*args, **kwargs)
tests/runners.py:73: in _run
    return klass(context).run(*args, **kwargs)
../../BUILDROOT/python-invoke-1.7.0-3.fc38.x86_64/usr/lib/python3.11/site-packages/invoke/runners.py:379: in run
    return self._run_body(command, **kwargs)
../../BUILDROOT/python-invoke-1.7.0-3.fc38.x86_64/usr/lib/python3.11/site-packages/invoke/runners.py:441: in _run_body
    return self.make_promise() if self._asynchronous else self._finish()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_util._Dummy object at 0x7f232f1883d0>

    def _finish(self):
        # Wait for subprocess to run, forwarding signals as we get them.
        try:
            while True:
                try:
                    self.wait()
                    break  # done waiting!
                # Don't locally stop on ^C, only forward it:
                # - if remote end really stops, we'll naturally stop after
                # - if remote end does not stop (eg REPL, editor) we don't want
                # to stop prematurely
                except KeyboardInterrupt as e:
                    self.send_interrupt(e)
                # TODO: honor other signals sent to our own process and
                # transmit them to the subprocess before handling 'normally'.
        # Make sure we tie off our worker threads, even if something exploded.
        # Any exceptions that raised during self.wait() above will appear after
        # this block.
        finally:
            # Inform stdin-mirroring worker to stop its eternal looping
            self.program_finished.set()
            # Join threads, storing inner exceptions, & set a timeout if
            # necessary. (Segregate WatcherErrors as they are "anticipated
            # errors" that want to show up at the end during creation of
            # Failure objects.)
            watcher_errors = []
            thread_exceptions = []
            for target, thread in six.iteritems(self.threads):
                thread.join(self._thread_join_timeout(target))
                exception = thread.exception()
                if exception is not None:
                    real = exception.value
                    if isinstance(real, WatcherError):
                        watcher_errors.append(real)
                    else:
                        thread_exceptions.append(exception)
        # If any exceptions appeared inside the threads, raise them now as an
        # aggregate exception object.
        # NOTE: this is kept outside the 'finally' so that main-thread
        # exceptions are raised before worker-thread exceptions; they're more
        # likely to be Big Serious Problems.
        if thread_exceptions:
>           raise ThreadException(thread_exceptions)
E           invoke.exceptions.ThreadException: 
E           Saw 1 exceptions within threads (OSError):
E           
E           
E           Thread args: {'kwargs': {'echo': None,
E                       'input_': <_pytest.capture.DontReadFromInput object at 0x7f23302703d0>,
E                       'output': <pytest_relaxed.trap.CarbonCopy object at 0x7f232eee14e0>},
E            'target': <bound method Runner.handle_stdin of <_util._Dummy object at 0x7f232f1883d0>>}
E           
E           Traceback (most recent call last):
E           
E             File "/builddir/build/BUILDROOT/python-invoke-1.7.0-3.fc38.x86_64/usr/lib/python3.11/site-packages/invoke/util.py", line 237, in run
E               super(ExceptionHandlingThread, self).run()
E           
E             File "/usr/lib64/python3.11/threading.py", line 975, in run
E               self._target(*self._args, **self._kwargs)
E           
E             File "/builddir/build/BUILDROOT/python-invoke-1.7.0-3.fc38.x86_64/usr/lib/python3.11/site-packages/invoke/runners.py", line 834, in handle_stdin
E               data = self.read_our_stdin(input_)
E                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
E           
E             File "/builddir/build/BUILDROOT/python-invoke-1.7.0-3.fc38.x86_64/usr/lib/python3.11/site-packages/invoke/runners.py", line 793, in read_our_stdin
E               bytes_ = input_.read(bytes_to_read(input_))
E                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E           
E             File "/usr/lib/python3.11/site-packages/_pytest/capture.py", line 192, in read
E               raise OSError(
E           
E           OSError: pytest: reading from stdin while output is captured!  Consider using `-s`.

../../BUILDROOT/python-invoke-1.7.0-3.fc38.x86_64/usr/lib/python3.11/site-packages/invoke/runners.py:493: ThreadException
=========================== short test summary info ============================
FAILED tests/context.py::Context_::sudo::auto_response_merges_with_other_responses::kwarg_only_adds_to_kwarg
FAILED tests/context.py::Context_::sudo::auto_response_merges_with_other_responses::config_only
FAILED tests/context.py::Context_::sudo::auto_response_merges_with_other_responses::both_kwarg_and_config
FAILED tests/runners.py::Runner_::command_echoing::uses_custom_format - invok...
=======

Comment 10 Jiri Kucera 2022-10-04 19:32:54 UTC
It is on the road to rawhide: https://koji.fedoraproject.org/koji/taskinfo?taskID=92628723

Comment 11 Miro Hrončok 2022-10-04 19:37:51 UTC
Note that the tests failed but it did not stop the build:

+ expect -c 'spawn pytest-3 -s; expect default'
spawn pytest-3 -s
ImportError while loading conftest '/builddir/build/BUILD/invoke-1.7.0/tests/conftest.py'.
tests/conftest.py:14: in <module>
    from _util import support
tests/_util.py:20: in <module>
    from invoke import Program, Runner
E   ModuleNotFoundError: No module named 'invoke'


I suspect the invocation does not pass the non-zero exit code.

Comment 12 Miro Hrončok 2022-10-04 19:49:12 UTC
This makes the tests run:

 %check
-PYTHONDONTWRITEBYTECODE=1 \
-PYTHONPATH=%{buildroot}%{python3_sitelib} \
+export PYTHONDONTWRITEBYTECODE=1
+export PYTHONPATH=%{buildroot}%{python3_sitelib}


But I have no idea how to make %check fail if the tests fail.

Comment 13 Jiri Kucera 2022-10-04 20:38:45 UTC
I wrote explanatory comment in the wrong place *after* a successful scratch build, fixed now. I also added few lines to the expect script to propagate spawned process' exit code.
https://koji.fedoraproject.org/koji/taskinfo?taskID=92631369

Comment 14 Miro Hrončok 2022-10-04 21:15:01 UTC
Thanks. I verified it actually fails if pytest fails.

Leaving open to get the getfullargspec fix propagated to F37.

Comment 15 Fedora Update System 2022-10-05 10:45:46 UTC
FEDORA-2022-a017e4a880 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2022-a017e4a880

Comment 16 Fedora Update System 2022-10-05 10:47:32 UTC
FEDORA-2022-a017e4a880 has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 17 Dominik 'Rathann' Mierzejewski 2023-02-02 12:19:30 UTC
Can we get a fix in F37?

Comment 18 Dominik 'Rathann' Mierzejewski 2023-02-02 12:24:07 UTC
invoke is still failing in F37: https://koji.fedoraproject.org/koji/taskinfo?taskID=97002196 .

Comment 19 Fedora Update System 2023-02-06 00:04:26 UTC
FEDORA-2023-6491d37a85 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2023-6491d37a85

Comment 20 Jiri Kucera 2023-02-06 00:06:44 UTC
I forgot to build this after Fedora 37 release, sorry.

Comment 21 Fedora Update System 2023-02-06 02:39:10 UTC
FEDORA-2023-6491d37a85 has been pushed to the Fedora 37 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-6491d37a85`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-6491d37a85

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 22 Fedora Update System 2023-02-14 01:52:15 UTC
FEDORA-2023-6491d37a85 has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.