Bug 1598574

Summary: pyanaconda.dbus.typing broken by Python 3.7 ("TypeError: Unknown type: typing.Tuple[int, str]")
Product: [Fedora] Fedora Reporter: Adam Williamson <awilliam>
Component: anacondaAssignee: Anaconda Maintenance Team <anaconda-maint-list>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: rawhideCC: anaconda-maint-list, jonathan, kellin, kevin, mhroncok, robatino, vanmeeuwen+fedora, vponcova, wwoods
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard: AcceptedBlocker
Fixed In Version: anaconda-29.20-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-08-05 19:49:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1517011, 1565020    

Description Adam Williamson 2018-07-05 20:59:23 UTC
Since Python 3.7 was merged to Rawhide, live image creation is failing with this traceback:

    Traceback (most recent call last):
    File "/usr/sbin/anaconda", line 285, in <module>
    from pyanaconda import startup_utils
    File "/usr/lib64/python3.7/site-packages/pyanaconda/startup_utils.py", line 38, in <module>
    from pyanaconda import kickstart
    File "/usr/lib64/python3.7/site-packages/pyanaconda/kickstart.py", line 60, in <module>
    from pyanaconda.modules.common.task import sync_run_task
    File "/usr/lib64/python3.7/site-packages/pyanaconda/modules/common/task/__init__.py", line 21, in <module>
    from pyanaconda.modules.common.task.task_interface import TaskInterface
    File "/usr/lib64/python3.7/site-packages/pyanaconda/modules/common/task/task_interface.py", line 33, in <module>
    class TaskInterface(InterfaceTemplate):
    File "/usr/lib64/python3.7/site-packages/pyanaconda/dbus/interface.py", line 79, in decorated
    cls.dbus = generator.generate_specification(cls, interface_name)
    File "/usr/lib64/python3.7/site-packages/pyanaconda/dbus/interface.py", line 182, in generate_specification
    interface = self._generate_interface(cls, all_interfaces, interface_name)
    File "/usr/lib64/python3.7/site-packages/pyanaconda/dbus/interface.py", line 278, in _generate_interface
    element = self._generate_property(member, member_name)
    File "/usr/lib64/python3.7/site-packages/pyanaconda/dbus/interface.py", line 424, in _generate_property
    return self.xml_generator.create_property(member_name, get_dbus_type(type_hint), access)
    File "/usr/lib64/python3.7/site-packages/pyanaconda/dbus/typing.py", line 67, in get_dbus_type
    return DBusType.get_dbus_representation(type_hint)
    File "/usr/lib64/python3.7/site-packages/pyanaconda/dbus/typing.py", line 137, in get_dbus_representation
    raise TypeError("Unknown type: %s" % type_hint)
    TypeError: Unknown type: typing.Tuple[int, str]

Digging into it, this is ultimately why it's failing. Python 3.6:

>>> from typing import Tuple
>>> tup = Tuple[int, str]
>>> getattr(tup, "__origin__")
typing.Tuple

Python 3.7:

>>> from typing import Tuple
>>> tup = Tuple[int, str]
>>> getattr(tup, "__origin__")
<class 'tuple'>

pyanaconda.dbus.typing._is_container_type() expects this __origin__ attribute to *be* the class of the relevant hint:

    def _is_container_type(type_hint):
        """Is it a container type?"""
        # Try to get the "base" type of the container type.
        origin = getattr(type_hint, "__origin__", None)
        return origin in DBusType._container_type_mapping

as we can see above, this works in Python 3.6 - the __origin__ attribute of a typing.Tuple hint is the class typing.Tuple - but does *not* work in Python 3.7 - the __origin__ attribute of a typing.Tuple hint is the class tuple.

This is a compose-breaking issue, so marking it as an automatic Fedora 29 Beta blocker. Unfortunately, I can't see a very easy way to fix this, at least not yet - in Python 3.7 it seems almost impossible to take the object you get by doing `Tuple[int, str]` and work back from that object to the `typing.Tuple` class. The link just...isn't there. You could do it with some fairly horrible string comparison, but not *sure* I want to submit a patch for that.

Comment 1 Adam Williamson 2018-07-05 21:17:13 UTC
For the record, this is the hideous string comparison I came up with, that seems to work:

diff --git a/pyanaconda/dbus/typing.py b/pyanaconda/dbus/typing.py
index ea8a2999c..8915a421f 100644
--- a/pyanaconda/dbus/typing.py
+++ b/pyanaconda/dbus/typing.py
@@ -150,14 +150,15 @@ class DBusType(object):
     def _is_container_type(type_hint):
         """Is it a container type?"""
         # Try to get the "base" type of the container type.
-        origin = getattr(type_hint, "__origin__", None)
-        return origin in DBusType._container_type_mapping
+        return any(str(type_hint).startswith(str(hintcls))
+                   for hintcls in DBusType._container_type_mapping)
 
     @staticmethod
     def _get_container_type(type_hint):
         """Return a container type."""
         # Get the "base" type of the container.
-        origin = type_hint.__origin__
+        origin = [hintcls for hintcls in DBusType._container_type_mapping
+                  if str(type_hint).startswith(str(hintcls))][0]
         # Get the arguments of the container.
         args = type_hint.__args__
 

....ew.

Comment 3 Adam Williamson 2018-07-05 21:56:51 UTC
Someone at github suggested using this:

https://github.com/ilevkivskyi/typing_inspect

seems like its `get_origin()` would be what we want, but it's not yet packaged for Fedora.

Comment 4 Adam Williamson 2018-07-05 22:01:45 UTC
Well...actually, it looks like that doesn't support List or Dict, so not much use.

Comment 5 Miro Hrončok 2018-07-06 11:11:07 UTC
$ python3.6
>>> from typing import Tuple
>>> tup = Tuple[int, str]
>>> origin = getattr(tup, "__origin__")
>>> issubclass(origin, tuple)
True


$ python3.7
>>> from typing import Tuple
>>> tup = Tuple[int, str]
>>> origin = getattr(tup, "__origin__")
>>> issubclass(origin, tuple)
True


Would this be helpful?

E.g. instead of:

return origin in DBusType._container_type_mapping

any(issubclass(origin, c) for c in DBusType._container_type_mapping)

Comment 6 Miro Hrončok 2018-07-06 11:14:22 UTC
note that aslo this is true on both versions:

>>> issubclass(origin, Tuple)
True

Comment 7 Miro Hrončok 2018-07-06 11:18:17 UTC
And:

     @staticmethod
     def _get_container_type(type_hint):
         """Return a container type."""
         # Get the "base" type of the container.
-        origin = type_hint.__origin__
+        origin = tuple(c for c in DBusType._container_type_mapping if issubclass(type_hint.__origin__, c))[0]
         # Get the arguments of the container.
         args = type_hint.__args__

Sorry, I'm from cell phone data only, on my laptop, outside (it's a public holiday here) so I won't be able to try it out.

It's a bit ugly, but at least it's not string matching.

Comment 8 Miro Hrončok 2018-07-06 11:31:32 UTC
Note that the tuple comprehension might be replaced with a generator and 1 call to next, if we want to make it more efficient. I have no idea how long the dbus type list is.

Comment 9 Miro Hrončok 2018-07-06 11:40:30 UTC
any(issubclass(origin, c) for c in DBusType._container_type_mapping)

Might be easier to just:

issubclass(origin, DBusType._container_type_mapping)

Comment 10 Miro Hrončok 2018-07-06 11:43:59 UTC
Well, DBusType._container_type_mapping must be (converted to) a tuple for that to actually work.

Comment 11 Adam Williamson 2018-07-06 22:19:25 UTC
Thanks, Miro - that's the suggestion I like most so far. I've sent a PR based on it:

https://github.com/rhinstaller/anaconda/pull/1526

I'm going to do an anaconda build with that PR patched in, as we really want to try and get a compose with Python 3.7 through some time soon.

Comment 12 Miro Hrončok 2018-07-07 00:06:25 UTC
Is there anything that could have been done to prevent this kind of breakage? E.g. is it possible to build a "scratch" compose from a side tag to see if it works? If so, we can set it as a goal for next time (Python 3.8).

Comment 13 Adam Williamson 2018-07-07 07:24:31 UTC
nirik should know the answer to that, I'm not sure off the top of my head whether that's possible or not.

Comment 14 Kevin Fenzi 2018-07-07 20:14:41 UTC
There isn't currently, but we really want there to be. For just such things as this.

Comment 15 Miro Hrončok 2018-07-07 20:27:47 UTC
Thanks for the info. 3.8 is couple releases away, so hopefully, it'll be possible by then.


BTW latest compose says DOOMED but there are os many logs that I don't know where to look for the cause to see if it is related to this bug or something else.

Comment 16 Kevin Fenzi 2018-07-07 22:21:08 UTC
It's something else currently. bug 1599009 (cdrkit changes broke k3b that broke the kde spin).

Comment 17 Adam Williamson 2018-07-09 16:26:32 UTC
You're most likely to find discussion of failed composes at https://pagure.io/dusty/failed-composes/issues and/or on #fedora-releng on IRC, FWIW.

Comment 18 Adam Williamson 2018-07-10 19:14:35 UTC
Note this is fixed for current composes by downstream patch of my PR, but the PR is still open upstream. So leaving the bug open for now.

Comment 19 Vendula Poncova 2018-07-18 15:40:58 UTC
Fixed in a pull request that is based on a solution from Adam:
https://github.com/rhinstaller/anaconda/pull/1538

Comment 20 Miro Hrončok 2018-08-05 19:49:06 UTC
The PR is merged and this has been patched in rawhide long ago.