Bug 1964955

Summary: pyproj fails to build with Python 3.10: TypeError: invalid Flag 'GeodIntermediateFlag' -- missing values: 4, 8, 32, 64, 128, 512, 1024, 2048
Product: [Fedora] Fedora Reporter: Tomáš Hrnčiar <thrnciar>
Component: pyprojAssignee: Jos de Kloe <josdekloe>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: e, josdekloe, mhroncok, thrnciar
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-26 19:53:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1890881    

Description Tomáš Hrnčiar 2021-05-26 12:24:24 UTC
pyproj fails to build with Python 3.10.0b1.

I am not sure if this is connected with Python 3.10, but scratch build on rawhide passed. 

Configuration error:
There is a programmable error in your configuration file:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/sphinx/config.py", line 327, in eval_config_file
    execfile_(filename, namespace)
  File "/usr/lib/python3.10/site-packages/sphinx/util/pycompat.py", line 88, in execfile_
    exec(code, _globals)
  File "/builddir/build/BUILD/pyproj-3.1.0/docs/conf.py", line 23, in <module>
    import pyproj
  File "/builddir/build/BUILD/pyproj-3.1.0/build/lib.linux-x86_64-3.10/pyproj/__init__.py", line 55, in <module>
    from pyproj.crs import CRS  # noqa: F401
  File "/builddir/build/BUILD/pyproj-3.1.0/build/lib.linux-x86_64-3.10/pyproj/crs/__init__.py", line 7, in <module>
    from pyproj._crs import (  # noqa: F401
  File "pyproj/_crs.pyx", line 12, in init pyproj._crs
  File "/builddir/build/BUILD/pyproj-3.1.0/build/lib.linux-x86_64-3.10/pyproj/crs/enums.py", line 4, in <module>
    from pyproj.enums import BaseEnum
  File "/builddir/build/BUILD/pyproj-3.1.0/build/lib.linux-x86_64-3.10/pyproj/enums.py", line 141, in <module>
    class GeodIntermediateFlag(IntFlag):
  File "/usr/lib64/python3.10/enum.py", line 544, in __new__
    raise TypeError(
TypeError: invalid Flag 'GeodIntermediateFlag' -- missing values: 4, 8, 32, 64, 128, 512, 1024, 2048

For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.10/fedora-rawhide-x86_64/02207050-pyproj/

For all our attempts to build pyproj with Python 3.10, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.10/package/pyproj/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.10:
https://copr.fedorainfracloud.org/coprs/g/python/python3.10/

Let us know here if you have any questions.

Python 3.10 will be included in Fedora 35. To make that update smoother, we're building Fedora packages with early pre-releases of Python 3.10.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 1 Jos de Kloe 2021-05-26 19:07:10 UTC
Hi Tomáš,

thanks a lot for reporting this issue.
Using the COPR test possibility that you provided I could reproduce the problem on my local machine.

I think this is caused by some undocumented behaviour in python 3.10.0b1 (or maybe I could just not find it in the documentation or maybe it is a bug in this python version?).
Anyway, thanks to the COPR test possibility I could quickly zoom in to the problem. It seems that it is no longer allowed in this python version to define an enum.IntFlag subclass with a sequence of non consecutive bit values. And this is how IntFlag is used at the moment in the pyproj/enum.py module. If I add dummy flag constants to ensure the full range of bits between the minimum and maximum used value is defined then the code runs just fine again, also for python 3.10.0b1, in my local mock test.

I will now kick off a koji build for regular rawhide to ensure I did not break anything there.

Comment 2 Miro Hrončok 2021-05-26 19:15:59 UTC
This change seem to have introduced the behavior: https://github.com/python/cpython/commit/7aaeb2a3d682ecba125c33511e4b4796021d2f82

Comment 3 Miro Hrončok 2021-05-26 19:32:42 UTC
I've reported https://bugs.python.org/issue44242

Comment 4 Jos de Kloe 2021-05-26 19:36:39 UTC
The koji build in rawhide succeeded for python 3.9.5, so you can try to re-build pyproj now in COPR.

Miro, thanks for reporting this to bugs.python.org.
From your answer I get that this is actual new behavior for python3.10.0b1 and not a bug in itself, but more a lack in the documentation, so I will inform upstream about this.

From the change in git it seems that another way to solve this would be to use something like:

class GeodIntermediateFlag(IntFlag, boundary=enum.KEEP):

in stead of:

class GeodIntermediateFlag(IntFlag):

But actually I am not sure if I like that. I find that a lot less obvious to understand when just reading the code.

Comment 5 Miro Hrončok 2021-05-26 19:53:15 UTC
Also, boundary=enum.KEEP is not yet available in Python <= 3.9, so you would need to use something a bit far-fetched for compatibility, e.g.:

try:
    from enum import IntFlag, KEEP
    INTFLAG_KWARGS = {'boundary': KEEP}
except ImportError:
    INTFLAG_KWARGS = {}


class GeodIntermediateFlag(IntFlag, **INTFLAG_KWARGS):
    ...


I would not jump into conclusions whether or not this is a bug or just missing documentation. Backwards incompatible changes in Python have rules and boundary=enum.KEEP might just as well become the default.



However, your solution with UNUSED enum values gets the job done, so I'll close this as fixed. Thanks!