Bug 1598574 - pyanaconda.dbus.typing broken by Python 3.7 ("TypeError: Unknown type: typing.Tuple[int, str]")
Summary: pyanaconda.dbus.typing broken by Python 3.7 ("TypeError: Unknown type: typing...
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: anaconda
Version: rawhide
Hardware: All
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Anaconda Maintenance Team
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedBlocker
Depends On:
Blocks: F29BetaBlocker PYTHON37
TreeView+ depends on / blocked
 
Reported: 2018-07-05 20:59 UTC by Adam Williamson
Modified: 2018-08-05 19:49 UTC (History)
9 users (show)

Fixed In Version: anaconda-29.20-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-08-05 19:49:06 UTC


Attachments (Terms of Use)

Description Adam Williamson 2018-07-05 20:59:23 UTC
Since Python 3.7 was merged to Rawhide, live image creation is failing with this traceback:

    Traceback (most recent call last):
    File "/usr/sbin/anaconda", line 285, in <module>
    from pyanaconda import startup_utils
    File "/usr/lib64/python3.7/site-packages/pyanaconda/startup_utils.py", line 38, in <module>
    from pyanaconda import kickstart
    File "/usr/lib64/python3.7/site-packages/pyanaconda/kickstart.py", line 60, in <module>
    from pyanaconda.modules.common.task import sync_run_task
    File "/usr/lib64/python3.7/site-packages/pyanaconda/modules/common/task/__init__.py", line 21, in <module>
    from pyanaconda.modules.common.task.task_interface import TaskInterface
    File "/usr/lib64/python3.7/site-packages/pyanaconda/modules/common/task/task_interface.py", line 33, in <module>
    class TaskInterface(InterfaceTemplate):
    File "/usr/lib64/python3.7/site-packages/pyanaconda/dbus/interface.py", line 79, in decorated
    cls.dbus = generator.generate_specification(cls, interface_name)
    File "/usr/lib64/python3.7/site-packages/pyanaconda/dbus/interface.py", line 182, in generate_specification
    interface = self._generate_interface(cls, all_interfaces, interface_name)
    File "/usr/lib64/python3.7/site-packages/pyanaconda/dbus/interface.py", line 278, in _generate_interface
    element = self._generate_property(member, member_name)
    File "/usr/lib64/python3.7/site-packages/pyanaconda/dbus/interface.py", line 424, in _generate_property
    return self.xml_generator.create_property(member_name, get_dbus_type(type_hint), access)
    File "/usr/lib64/python3.7/site-packages/pyanaconda/dbus/typing.py", line 67, in get_dbus_type
    return DBusType.get_dbus_representation(type_hint)
    File "/usr/lib64/python3.7/site-packages/pyanaconda/dbus/typing.py", line 137, in get_dbus_representation
    raise TypeError("Unknown type: %s" % type_hint)
    TypeError: Unknown type: typing.Tuple[int, str]

Digging into it, this is ultimately why it's failing. Python 3.6:

>>> from typing import Tuple
>>> tup = Tuple[int, str]
>>> getattr(tup, "__origin__")
typing.Tuple

Python 3.7:

>>> from typing import Tuple
>>> tup = Tuple[int, str]
>>> getattr(tup, "__origin__")
<class 'tuple'>

pyanaconda.dbus.typing._is_container_type() expects this __origin__ attribute to *be* the class of the relevant hint:

    def _is_container_type(type_hint):
        """Is it a container type?"""
        # Try to get the "base" type of the container type.
        origin = getattr(type_hint, "__origin__", None)
        return origin in DBusType._container_type_mapping

as we can see above, this works in Python 3.6 - the __origin__ attribute of a typing.Tuple hint is the class typing.Tuple - but does *not* work in Python 3.7 - the __origin__ attribute of a typing.Tuple hint is the class tuple.

This is a compose-breaking issue, so marking it as an automatic Fedora 29 Beta blocker. Unfortunately, I can't see a very easy way to fix this, at least not yet - in Python 3.7 it seems almost impossible to take the object you get by doing `Tuple[int, str]` and work back from that object to the `typing.Tuple` class. The link just...isn't there. You could do it with some fairly horrible string comparison, but not *sure* I want to submit a patch for that.

Comment 1 Adam Williamson 2018-07-05 21:17:13 UTC
For the record, this is the hideous string comparison I came up with, that seems to work:

diff --git a/pyanaconda/dbus/typing.py b/pyanaconda/dbus/typing.py
index ea8a2999c..8915a421f 100644
--- a/pyanaconda/dbus/typing.py
+++ b/pyanaconda/dbus/typing.py
@@ -150,14 +150,15 @@ class DBusType(object):
     def _is_container_type(type_hint):
         """Is it a container type?"""
         # Try to get the "base" type of the container type.
-        origin = getattr(type_hint, "__origin__", None)
-        return origin in DBusType._container_type_mapping
+        return any(str(type_hint).startswith(str(hintcls))
+                   for hintcls in DBusType._container_type_mapping)
 
     @staticmethod
     def _get_container_type(type_hint):
         """Return a container type."""
         # Get the "base" type of the container.
-        origin = type_hint.__origin__
+        origin = [hintcls for hintcls in DBusType._container_type_mapping
+                  if str(type_hint).startswith(str(hintcls))][0]
         # Get the arguments of the container.
         args = type_hint.__args__
 

....ew.

Comment 3 Adam Williamson 2018-07-05 21:56:51 UTC
Someone at github suggested using this:

https://github.com/ilevkivskyi/typing_inspect

seems like its `get_origin()` would be what we want, but it's not yet packaged for Fedora.

Comment 4 Adam Williamson 2018-07-05 22:01:45 UTC
Well...actually, it looks like that doesn't support List or Dict, so not much use.

Comment 5 Miro Hrončok 2018-07-06 11:11:07 UTC
$ python3.6
>>> from typing import Tuple
>>> tup = Tuple[int, str]
>>> origin = getattr(tup, "__origin__")
>>> issubclass(origin, tuple)
True


$ python3.7
>>> from typing import Tuple
>>> tup = Tuple[int, str]
>>> origin = getattr(tup, "__origin__")
>>> issubclass(origin, tuple)
True


Would this be helpful?

E.g. instead of:

return origin in DBusType._container_type_mapping

any(issubclass(origin, c) for c in DBusType._container_type_mapping)

Comment 6 Miro Hrončok 2018-07-06 11:14:22 UTC
note that aslo this is true on both versions:

>>> issubclass(origin, Tuple)
True

Comment 7 Miro Hrončok 2018-07-06 11:18:17 UTC
And:

     @staticmethod
     def _get_container_type(type_hint):
         """Return a container type."""
         # Get the "base" type of the container.
-        origin = type_hint.__origin__
+        origin = tuple(c for c in DBusType._container_type_mapping if issubclass(type_hint.__origin__, c))[0]
         # Get the arguments of the container.
         args = type_hint.__args__

Sorry, I'm from cell phone data only, on my laptop, outside (it's a public holiday here) so I won't be able to try it out.

It's a bit ugly, but at least it's not string matching.

Comment 8 Miro Hrončok 2018-07-06 11:31:32 UTC
Note that the tuple comprehension might be replaced with a generator and 1 call to next, if we want to make it more efficient. I have no idea how long the dbus type list is.

Comment 9 Miro Hrončok 2018-07-06 11:40:30 UTC
any(issubclass(origin, c) for c in DBusType._container_type_mapping)

Might be easier to just:

issubclass(origin, DBusType._container_type_mapping)

Comment 10 Miro Hrončok 2018-07-06 11:43:59 UTC
Well, DBusType._container_type_mapping must be (converted to) a tuple for that to actually work.

Comment 11 Adam Williamson 2018-07-06 22:19:25 UTC
Thanks, Miro - that's the suggestion I like most so far. I've sent a PR based on it:

https://github.com/rhinstaller/anaconda/pull/1526

I'm going to do an anaconda build with that PR patched in, as we really want to try and get a compose with Python 3.7 through some time soon.

Comment 12 Miro Hrončok 2018-07-07 00:06:25 UTC
Is there anything that could have been done to prevent this kind of breakage? E.g. is it possible to build a "scratch" compose from a side tag to see if it works? If so, we can set it as a goal for next time (Python 3.8).

Comment 13 Adam Williamson 2018-07-07 07:24:31 UTC
nirik should know the answer to that, I'm not sure off the top of my head whether that's possible or not.

Comment 14 Kevin Fenzi 2018-07-07 20:14:41 UTC
There isn't currently, but we really want there to be. For just such things as this.

Comment 15 Miro Hrončok 2018-07-07 20:27:47 UTC
Thanks for the info. 3.8 is couple releases away, so hopefully, it'll be possible by then.


BTW latest compose says DOOMED but there are os many logs that I don't know where to look for the cause to see if it is related to this bug or something else.

Comment 16 Kevin Fenzi 2018-07-07 22:21:08 UTC
It's something else currently. bug 1599009 (cdrkit changes broke k3b that broke the kde spin).

Comment 17 Adam Williamson 2018-07-09 16:26:32 UTC
You're most likely to find discussion of failed composes at https://pagure.io/dusty/failed-composes/issues and/or on #fedora-releng on IRC, FWIW.

Comment 18 Adam Williamson 2018-07-10 19:14:35 UTC
Note this is fixed for current composes by downstream patch of my PR, but the PR is still open upstream. So leaving the bug open for now.

Comment 19 Vendula Poncova 2018-07-18 15:40:58 UTC
Fixed in a pull request that is based on a solution from Adam:
https://github.com/rhinstaller/anaconda/pull/1538

Comment 20 Miro Hrončok 2018-08-05 19:49:06 UTC
The PR is merged and this has been patched in rawhide long ago.


Note You need to log in before you can comment on or make changes to this bug.