Bug 2329164 - Package FTBFS with new Python
Summary: Package FTBFS with new Python
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: gdal
Version: rawhide
Hardware: Unspecified
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Sandro Mani
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: F42FTBFS 2330189
TreeView+ depends on / blocked
 
Reported: 2024-11-27 15:04 UTC by Richard W.M. Jones
Modified: 2024-12-03 21:24 UTC (History)
8 users (show)

Fixed In Version: gdal-3.10.0-4.fc42
Clone Of:
Environment:
Last Closed: 2024-12-03 21:24:26 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Richard W.M. Jones 2024-11-27 15:04:48 UTC
See:
https://koji.fedoraproject.org/koji/taskinfo?taskID=126300839

This seems to be related to the newer version of Python.  There are pages
and pages of Python errors in the swig-generated bindings.  It's unclear
if there is a single root cause or not.

Reproducible: Always

Steps to Reproduce:
1. Rebuild the package

Comment 1 Ben Beasley 2024-11-27 17:07:19 UTC
This seems to be MinGW-related. The first failing build in Koschei was https://koschei.fedoraproject.org/build/19071285.

Comment 2 Sandro Mani 2024-11-27 22:44:54 UTC
Looks related to mingw-python3-3.11.10

Comment 3 Elliott Sales de Andrade 2024-11-28 01:52:30 UTC
None of this extension linking looks right; they aren't linking to Python:

i686-w64-mingw32-gcc -shared -Wl,--enable-auto-image-base -fstack-protector -lssp -fstack-protector -lssp -fstack-protector -lssp -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions --param=ssp-buffer-size=4 build/temp.mingw32-cpython-311/extensions/gdalconst_wrap.o -L/builddir/build/BUILD/gdal-3.10.0-build/gdal-3.10.0-fedora/build_win32 -L/usr/i686-w64-mingw32/sys-root/mingw/lib -lgdal -o build/lib.mingw32-cpython-311/osgeo/_gdalconst.cpython-311.dll
i686-w64-mingw32-g++ -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions --param=ssp-buffer-size=4 -shared -Wl,--enable-auto-image-base -fstack-protector -lssp -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions --param=ssp-buffer-size=4 build/temp.mingw32-cpython-311/extensions/gnm_wrap.o -L/builddir/build/BUILD/gdal-3.10.0-build/gdal-3.10.0-fedora/build_win32 -L/usr/i686-w64-mingw32/sys-root/mingw/lib -lgdal -o build/lib.mingw32-cpython-311/osgeo/_gnm.cpython-311.dll
i686-w64-mingw32-g++ -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions --param=ssp-buffer-size=4 -shared -Wl,--enable-auto-image-base -fstack-protector -lssp -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions --param=ssp-buffer-size=4 build/temp.mingw32-cpython-311/extensions/gdal_array_wrap.o -L/builddir/build/BUILD/gdal-3.10.0-build/gdal-3.10.0-fedora/build_win32 -L/usr/i686-w64-mingw32/sys-root/mingw/lib -lgdal -o build/lib.mingw32-cpython-311/osgeo/_gdal_array.cpython-311.dll
i686-w64-mingw32-g++ -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions --param=ssp-buffer-size=4 -shared -Wl,--enable-auto-image-base -fstack-protector -lssp -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions --param=ssp-buffer-size=4 build/temp.mingw32-cpython-311/extensions/osr_wrap.o -L/builddir/build/BUILD/gdal-3.10.0-build/gdal-3.10.0-fedora/build_win32 -L/usr/i686-w64-mingw32/sys-root/mingw/lib -lgdal -o build/lib.mingw32-cpython-311/osgeo/_osr.cpython-311.dll
i686-w64-mingw32-g++ -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions --param=ssp-buffer-size=4 -shared -Wl,--enable-auto-image-base -fstack-protector -lssp -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions --param=ssp-buffer-size=4 build/temp.mingw32-cpython-311/extensions/ogr_wrap.o -L/builddir/build/BUILD/gdal-3.10.0-build/gdal-3.10.0-fedora/build_win32 -L/usr/i686-w64-mingw32/sys-root/mingw/lib -lgdal -o build/lib.mingw32-cpython-311/osgeo/_ogr.cpython-311.dll
i686-w64-mingw32-g++ -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions --param=ssp-buffer-size=4 -shared -Wl,--enable-auto-image-base -fstack-protector -lssp -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions --param=ssp-buffer-size=4 build/temp.mingw32-cpython-311/extensions/gdal_wrap.o -L/builddir/build/BUILD/gdal-3.10.0-build/gdal-3.10.0-fedora/build_win32 -L/usr/i686-w64-mingw32/sys-root/mingw/lib -lgdal -o build/lib.mingw32-cpython-311/osgeo/_gdal.cpython-311.dll

Going up to the CMake invocation, it doesn't seem to properly find Python:

-- Found SWIG: /usr/bin/swig (found version "4.3.0")
-- Could NOT find Python (missing: Python_NumPy_INCLUDE_DIRS NumPy) (found suitable version "3.11.10", minimum required is "3.8")

and later:

-- The following OPTIONAL packages have not been found:
 * Python (required version >= 3.8)
   SWIG_PYTHON: Python binding

but doesn't correctly disable/error.

Comment 4 Elliott Sales de Andrade 2024-11-28 04:13:59 UTC
Though also to note that on Linux-y systems, Python extensions are not supposed to have linked with its library. That can't work on Windows though as all symbols must be resolved IIRC.

And the build is pulling in mingw{32,64}-python3-numpy, but perhaps CMake needs a hint of some sort when cross-compiling (by setting some variable from https://cmake.org/cmake/help/git-stage/module/FindPython.html perhaps?)

Comment 5 Elliott Sales de Andrade 2024-11-28 06:52:56 UTC
For MSVC, Python headers include a pragma that adds the correct library linkage, but this doesn't apply to gcc:
https://github.com/python/cpython/blob/3a77980002845c22e5b294ca47a12d62bf5baf53/PC/pyconfig.h.in#L310-L336

MSYS2 has the following patch which adds Python linkage to extensions that we haven't been applying:
https://github.com/msys2/MINGW-packages/blob/master/mingw-w64-python/0044-pkg-config-windows-must-link-ext-with-python-lib.patch

As well as far too many other patches that I didn't go through, that we don't appear to be using:
https://github.com/msys2/MINGW-packages/tree/master/mingw-w64-python
but I have no idea if any of those would be needed.

However, GDAL uses CMake's FindPython module, so I'm not sure if it bypasses all of that.


For NumPy, it looks like CMake tries to run some code to fetch the include path:
https://gitlab.kitware.com/cmake/cmake/-/blob/master/Modules/FindPython/Support.cmake?ref_type=heads#L3913-3927
However, earlier on it checks Python_NumPy_INCLUDE_DIR:
https://gitlab.kitware.com/cmake/cmake/-/blob/master/Modules/FindPython/Support.cmake?ref_type=heads#L3891

So if you can set something like:
-DPython_NumPy_INCLUDE_DIR=/usr/i686-w64-mingw32/sys-root/mingw/lib/python3.11/site-packages/numpy/core/include/
the NumPy search will work (except that I don't know how to set that variable with a different path for mingw64 since everything is wrapped in the architecture-agnostic %mingw_cmake macro.)

Comment 6 Richard W.M. Jones 2024-11-28 11:04:35 UTC
Looking at the macros in /usr/lib/rpm/macros.d/macros.mingw/ and
/usr/lib/rpm/macros.d/macros.mingw*, I believe you
should be able to set:

export MINGW32_CMAKE_ARGS="-DPython_NumPy_INCLUDE_DIR=%{mingw32_libdir}/python3.11/site-packages/numpy/core/include/"
export MINGW64_CMAKE_ARGS="-DPython_NumPy_INCLUDE_DIR=%{mingw64_libdir}/python3.11/site-packages/numpy/core/include/"
export UCRT64_CMAKE_ARGS="-DPython_NumPy_INCLUDE_DIR=%{ucrt64_libdir}/python3.11/site-packages/numpy/core/include/"
%{mingw_cmake}

It seems as if the xx_CMAKE_ARGS environment variables are intended for arch-specific
extra parameters to be passed through.

There may be other macros which hide the "python3.11/site-packages" part, but I'm not sure
what they would be.

Comment 7 Sandro Mani 2024-11-29 17:14:25 UTC
Unfortunately the

Could NOT find Python (missing: Python_NumPy_INCLUDE_DIRS NumPy) (found suitable version "3.11.10", minimum required is "3.8")

is not the cause - that message also appeared previously with successful builds. I'm trying to figure why -lpython3.11 disappeared from the link commandline.

Comment 8 Sandro Mani 2024-11-29 18:19:08 UTC
The culprit actually is mingw-python-setuptools which I updated to 74.1.3 a couple of weeks ago.

Comment 9 Fedora Update System 2024-12-03 21:22:08 UTC
FEDORA-2024-14fdac501b (gdal-3.10.0-4.fc42 and mingw-python-setuptools-74.1.3-2.fc42) has been submitted as an update to Fedora 42.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-14fdac501b

Comment 10 Fedora Update System 2024-12-03 21:24:26 UTC
FEDORA-2024-14fdac501b (gdal-3.10.0-4.fc42 and mingw-python-setuptools-74.1.3-2.fc42) has been pushed to the Fedora 42 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.