Bug 1814243

Summary: bodhi fails to build with Python 3.9: _dummy_thread and dummy_threading modules have been removed
Product: [Fedora] Fedora Reporter: Miro Hrončok <mhroncok>
Component: bodhiAssignee: Clement Verna <clems.verna>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: rawhideCC: aurelien, clems.verna, cstratak, infra-sig, jeremy, kevin, mhroncok, mplch, randy
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-24 18:28:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1831336, 1833579    
Bug Blocks: 1785415    

Description Miro Hrončok 2020-03-17 13:18:22 UTC
bodhi fails to build with Python 3.9.0a4:

==================================== ERRORS ====================================
__________ ERROR collecting bodhi/tests/server/tasks/test_composer.py __________
ImportError while importing test module '/builddir/build/BUILD/bodhi-5.1.1/bodhi/tests/server/tasks/test_composer.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
bodhi/tests/server/tasks/test_composer.py:22: in <module>
    import dummy_threading
E   ModuleNotFoundError: No module named 'dummy_threading'

See https://docs.python.org/3.9/whatsnew/3.9.html#removed

"_dummy_thread and dummy_threading modules have been removed. These modules were deprecated since Python 3.7 which requires threading support."


For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.9/fedora-rawhide-x86_64/01309575-bodhi/

For all our attempts to build bodhi with Python 3.9, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/package/bodhi/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.9:
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/

Let us know here if you have any questions.

Python 3.9 will be included in Fedora 33. To make that update smoother, we're building Fedora packages with early pre-releases of Python 3.9.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.






Setting the severity to high.

This package is part of the initial bootstrap sequence. Without it, we cannot proceed with the bootstrap in a Koji side tag.

https://fedoraproject.org/wiki/Changes/Python3.9#Important_dates_and_plan

The current plan is to follow the "ideal point when we can start rebuilding in Koji" -- that is we need to get this bug fixed approximately in 2 months.

That includes potential uncovered bugs in packages that depend on this one.

Please acknowledge that you have read this message and that you can dedicate time to fix it. If you know already that you won't be able to fix it by the deadline, please let us know ASAP, so we can allocate resources to do that. Thank You.

Comment 1 Miro Hrončok 2020-04-05 23:17:41 UTC
Please acknowledge that you have read this message and that you can dedicate time to fix it. If you know already that you won't be able to fix it by the deadline, please let us know ASAP, so we can allocate resources to do that. Thank You.

Comment 2 Kevin Fenzi 2020-04-07 21:09:53 UTC
I am not sure. :) I am adding cverna here who is taking on bodhi support...

Comment 3 Clement Verna 2020-04-08 18:32:17 UTC
I will be looking at fixing this in bodhi upstream.

Comment 4 Miro Hrončok 2020-05-08 23:06:56 UTC
I've done:

# bodhi/tests/server/tasks/test_composer.py is Python 3.9 incompatible
# https://github.com/fedora-infra/bodhi/issues/4007
.test-venv/bin/python3 /usr/bin/py.test-3 --cov-fail-under=%{cov_fail_under} -v bodhi/tests --ignore bodhi/tests/server/tasks/test_composer.py

It uncovered more problems:

https://copr.fedorainfracloud.org/coprs/g/python/python3.9/build/1373946/

There is bz1831336 and others:

>       c = bindings.BodhiClient()
bodhi/tests/client/test___init__.py:849: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
bodhi/client/bindings.py:206: in __init__
    super(BodhiClient, self).__init__(base_url, login_url=base_url + 'login', username=username,
/usr/lib/python3.9/site-packages/fedora/client/openidbaseclient.py:181: in __init__
    self._load_cookies()
/usr/lib/python3.9/site-packages/fedora/client/openidbaseclient.py:317: in _load_cookies
    data = json.loads(f.read(), encoding='utf-8')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
s = '{}', cls = <class 'json.decoder.JSONDecoder'>, object_hook = None
parse_float = None, parse_int = None, parse_constant = None
object_pairs_hook = None, kw = {'encoding': 'utf-8'}
    def loads(s, *, cls=None, object_hook=None, parse_float=None,
            parse_int=None, parse_constant=None, object_pairs_hook=None, **kw):
        """Deserialize ``s`` (a ``str``, ``bytes`` or ``bytearray`` instance
        containing a JSON document) to a Python object.
    
        ``object_hook`` is an optional function that will be called with the
        result of any object literal decode (a ``dict``). The return value of
        ``object_hook`` will be used instead of the ``dict``. This feature
        can be used to implement custom decoders (e.g. JSON-RPC class hinting).
    
        ``object_pairs_hook`` is an optional function that will be called with the
        result of any object literal decoded with an ordered list of pairs.  The
        return value of ``object_pairs_hook`` will be used instead of the ``dict``.
        This feature can be used to implement custom decoders.  If ``object_hook``
        is also defined, the ``object_pairs_hook`` takes priority.
    
        ``parse_float``, if specified, will be called with the string
        of every JSON float to be decoded. By default this is equivalent to
        float(num_str). This can be used to use another datatype or parser
        for JSON floats (e.g. decimal.Decimal).
    
        ``parse_int``, if specified, will be called with the string
        of every JSON int to be decoded. By default this is equivalent to
        int(num_str). This can be used to use another datatype or parser
        for JSON integers (e.g. float).
    
        ``parse_constant``, if specified, will be called with one of the
        following strings: -Infinity, Infinity, NaN.
        This can be used to raise an exception if invalid JSON numbers
        are encountered.
    
        To use a custom ``JSONDecoder`` subclass, specify it with the ``cls``
        kwarg; otherwise ``JSONDecoder`` is used.
        """
        if isinstance(s, str):
            if s.startswith('\ufeff'):
                raise JSONDecodeError("Unexpected UTF-8 BOM (decode using utf-8-sig)",
                                      s, 0)
        else:
            if not isinstance(s, (bytes, bytearray)):
                raise TypeError(f'the JSON object must be str, bytes or bytearray, '
                                f'not {s.__class__.__name__}')
            s = s.decode(detect_encoding(s), 'surrogatepass')
    
        if (cls is None and object_hook is None and
                parse_int is None and parse_float is None and
                parse_constant is None and object_pairs_hook is None and not kw):
            return _default_decoder.decode(s)
        if cls is None:
            cls = JSONDecoder
        if object_hook is not None:
            kw['object_hook'] = object_hook
        if object_pairs_hook is not None:
            kw['object_pairs_hook'] = object_pairs_hook
        if parse_float is not None:
            kw['parse_float'] = parse_float
        if parse_int is not None:
            kw['parse_int'] = parse_int
        if parse_constant is not None:
            kw['parse_constant'] = parse_constant
>       return cls(**kw).decode(s)
E       TypeError: __init__() got an unexpected keyword argument 'encoding'
/usr/lib64/python3.9/json/__init__.py:359: TypeError


Looks like a bug in python-fedora.

Comment 5 Miro Hrončok 2020-05-19 15:00:21 UTC
BTW There is another failure with Python 3.9.0b1, when generating documentation:

error while formatting arguments for bodhi.client.bindings.BodhiClient.candidates: type object 'Iterable' has no attribute '_special'

I don't yet really understand that error, because I don't see anything like _special in bodhi's code.

Comment 6 Miro Hrončok 2020-05-19 15:19:58 UTC
(In reply to Miro Hrončok from comment #5)
> BTW There is another failure with Python 3.9.0b1, when generating
> documentation:
> 
> error while formatting arguments for
> bodhi.client.bindings.BodhiClient.candidates: type object 'Iterable' has no
> attribute '_special'
> 
> I don't yet really understand that error, because I don't see anything like
> _special in bodhi's code.

That is a Sphinx thing possibly fixed in https://github.com/sphinx-doc/sphinx/commit/46372726de435cbc6b70c99d95b7986f76563df4

Comment 7 Miro Hrončok 2020-05-19 22:14:57 UTC
(In reply to Miro Hrončok from comment #6)
> (In reply to Miro Hrončok from comment #5)
> > BTW There is another failure with Python 3.9.0b1, when generating
> > documentation:
> > 
> > error while formatting arguments for
> > bodhi.client.bindings.BodhiClient.candidates: type object 'Iterable' has no
> > attribute '_special'
> > 
> > I don't yet really understand that error, because I don't see anything like
> > _special in bodhi's code.
> 
> That is a Sphinx thing possibly fixed in
> https://github.com/sphinx-doc/sphinx/commit/
> 46372726de435cbc6b70c99d95b7986f76563df4

Fixed this one.