Bug 2155003

It seems more recent ax_python_devel [1] on upstream is required. It seems different path for platform specific code and pure python libraries is not handled well. Also change to configure or makefiles is needed. It seems setup.py is never called anywere in ldns code. It seems to me distutils were used only to obtain PYTHON_CPPFLAGS, PYTHON_LIBS and PYTHON_VERSION environment variables, no instructions from [2] covert that topic. But update of [1] does not help. It installs platform dependent library to wrong place on 64 bit system.

[1] https://www.gnu.org/software/autoconf-archive/ax_python_devel.html
[2] https://peps.python.org/pep-0632/

Tomáš, is where is ldns.py supposed to be installed? It imports _ldns.so into python and cannot work without native library. There are 3 files installed to work with python.

- _ldns.so - this obviously belongs to %python3_sitearch directory, because it is compiled native code
- ldns.py - swig generated file, which imports low-level _ldns.so library. I have chosen to install this to %python3_sitelib, but not sure it is correct.
- ldnsx.py - python only helper for more python-like work. I expect this belongs to %python3_sitelib as pure python code.

I am unsure where ldns.py should be installed. I did not find it in python guidelines. If the package composes from python-only module relying on native library, should both python and native library be installed into the same directory? Is it okay to keep them both in separate directories? Is there best practice for such situation? Or even give instructions where each part belongs?

Tomáš is on PTO, so I'll try to answer your questions.


> _ldns.so - this obviously belongs to %python3_sitearch directory, because it is compiled native code

Correct.

> ldns.py - swig generated file, which imports low-level _ldns.so library. I have chosen to install this to %python3_sitelib, but not sure it is correct.

It works but I would not consider that standard practice. Also, since this file will be in both .i686 and .x86_64 package and the bytecode cache (.pyc) might conflict, you would need to follow https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_Appendix/#_byte_compilation_reproducibility to avoid conflicts that could happen when the .pyc files are not bit-by-bit identical. I do not recommend installing that file in %python3_sitelib.

Python packages installed via Python standard tools install either to %python3_sitearch if they contain even one extension module (.so) or to %python3_sitelib if they are pure Python packages.

> ldnsx.py - python only helper for more python-like work. I expect this belongs to %python3_sitelib as pure python code.

Same as above.

> If the package composes from python-only module relying on native library, should both python and native library be installed into the same directory?

Yes.

> Is it okay to keep them both in separate directories?

Yes, but see above.

> Is there best practice for such situation?

Same directory.

> Or even give instructions where each part belongs?

If a single package installs an extension module, everything goes to %python3_sitearch which can be obtained from Python like this:

>>> import sysconfig
>>> sysconfig.get_path('platlib')

This path is '/usr/local/lib64/python3.11/site-packages' by default, but it is '/usr/lib64/python3.11/site-packages' when building RPM packages.

You can use this in your configure scripts instead of the distutils.sysconfig.get_python_lib(1,0) call.




-------------

The current layout seems to be:

$ repoquery --repo=rawhide -l python3-ldns-1.8.3-2.x86_64
...
/usr/lib64/python3.11/site-packages/__pycache__
/usr/lib64/python3.11/site-packages/__pycache__/ldns.cpython-311.opt-1.pyc
/usr/lib64/python3.11/site-packages/__pycache__/ldns.cpython-311.pyc
/usr/lib64/python3.11/site-packages/__pycache__/ldnsx.cpython-311.opt-1.pyc
/usr/lib64/python3.11/site-packages/__pycache__/ldnsx.cpython-311.pyc
/usr/lib64/python3.11/site-packages/_ldns.so
/usr/lib64/python3.11/site-packages/_ldns.so.3
/usr/lib64/python3.11/site-packages/_ldns.so.3.5.0
/usr/lib64/python3.11/site-packages/ldns
/usr/lib64/python3.11/site-packages/ldns.py
/usr/lib64/python3.11/site-packages/ldnsx.py


That seems to be worth keeping, but I noticed one problem and one weird thing.

1) The /usr/lib64/python3.11/site-packages/__pycache__ MUST not be owned by python3-ldns, see https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_explicit_lists or https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/#_files_to_include

2) Python extension modules are usually not so-versioned like .so.3.5.0. Instead, they use the Python extension suffix, e.g. .cpython-311-x86_64-linux-gnu.so (obtainable from sysconfig.get_config_var('EXT_SUFFIX'))

> It seems to me distutils were used only to obtain PYTHON_CPPFLAGS, PYTHON_LIBS and PYTHON_VERSION environment variables,

I don't know what those environment variables are supposed to be (probably some autotools thing), but all the calls to distutils.sysconfig.get_config_var() withing the configure script could probably be trivially replaced with sysconfig.get_config_var().

(In reply to Miro Hrončok from comment #4)
> > It seems to me distutils were used only to obtain PYTHON_CPPFLAGS, PYTHON_LIBS and PYTHON_VERSION environment variables,
> 
> I don't know what those environment variables are supposed to be (probably
> some autotools thing), but all the calls to
> distutils.sysconfig.get_config_var() withing the configure script could
> probably be trivially replaced with sysconfig.get_config_var().

I were confused by the history comments in /usr/share/aclocal/ax_python_devel.m4 file. Later I checked #serial has changed and they have already some support for sysconfig directly. With compatibility layer also for older pythons using distutils.

What confuses me variables storing results has changed after I updated that file from a package contents. So I tried to use the correct directories.
PYTHON_PLATFORM_SITE_PKG now exports python3_sitearch equivalent, where original PYTHON_SITE_PKG exports only python3_sitelib. After update of configure file it changed destination directories without any other change on ldns part. Which leads me to conclusion correct directory should be used. But not sure which it is supposed to be.

$ rpm -q python3-ldns
python3-ldns-1.8.1-7.fc37.x86_64

$ rpm -ql python3-ldns
/usr/lib/.build-id
/usr/lib/.build-id/79/454794ce336ff3fc46619722c08963db06e9e9
/usr/lib64/python3.11/site-packages/__pycache__
/usr/lib64/python3.11/site-packages/__pycache__/ldns.cpython-311.opt-1.pyc
/usr/lib64/python3.11/site-packages/__pycache__/ldns.cpython-311.pyc
/usr/lib64/python3.11/site-packages/__pycache__/ldnsx.cpython-311.opt-1.pyc
/usr/lib64/python3.11/site-packages/__pycache__/ldnsx.cpython-311.pyc
/usr/lib64/python3.11/site-packages/_ldns.so
/usr/lib64/python3.11/site-packages/_ldns.so.3
/usr/lib64/python3.11/site-packages/_ldns.so.3.2.0
/usr/lib64/python3.11/site-packages/ldns
/usr/lib64/python3.11/site-packages/ldns.py
/usr/lib64/python3.11/site-packages/ldnsx.py

It seems previous package installed everything into %python3_sitearch directory. Should that be kept or it should it be changed? As autoconf-archive package already contains working solution, I expect customization should be done only on ldns side. But I am not sure why it changed used directories. Is that an error or actual bug fix?

> It seems previous package installed everything into %python3_sitearch directory. Should that be kept or it should it be changed?

It should be kept.


> Is that an error or actual bug fix?

That depends on what the autoconf thing is supposed to do which I don't know. I can tell you what is the right thing to do for a Python package, but I cannot tell you what is the intended usage of some autoconf script, sorry.

Merged upstream, it should work. Built in rawhide.

Verified in https://copr.fedorainfracloud.org/coprs/g/python/python3.12/package/ldns/ that it builds now with Python 3.12.



I've eyeballed https://src.fedoraproject.org/rpms/ldns/c/0369f19e9ab16b9dd7e358b7c99f02733b4e8c94?branch=rawhide and I have some remarks.


1. The python3-ldns package now lists:
    %{python3_sitearch}/*
    %{python3_sitelib}/*

This means you misunderstood my advice (or decided to deliberately not follow it) and packaged some files to sitelib and some files to sitearch:

/usr/lib/python3.11/site-packages/__pycache__
/usr/lib/python3.11/site-packages/__pycache__/ldnsx.cpython-311.opt-1.pyc
/usr/lib/python3.11/site-packages/__pycache__/ldnsx.cpython-311.pyc
/usr/lib/python3.11/site-packages/ldns
/usr/lib/python3.11/site-packages/ldnsx.py
/usr/lib64/python3.11/site-packages/__pycache__
/usr/lib64/python3.11/site-packages/__pycache__/ldns.cpython-311.opt-1.pyc
/usr/lib64/python3.11/site-packages/__pycache__/ldns.cpython-311.pyc
/usr/lib64/python3.11/site-packages/_ldns.so
/usr/lib64/python3.11/site-packages/_ldns.so.3
/usr/lib64/python3.11/site-packages/_ldns.so.3.5.0
/usr/lib64/python3.11/site-packages/ldns.py

I wonder why.

Anyway, if you do it that way, the files in /usr/lib/python3.11/site-packages/__pycache__ now exist in both x86_64 and i686 versions of the package and they may or may not be bit-by-bit identical. You need to follow https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_Appendix/#_byte_compilation_reproducibility


2. The (nonexistent) python2-ldns package now lists:
    %{python2_sitearch}/*
    %{python3_sitelib}/*

This is probably a typo.


3. Listing %{python3_sitearch}/* and/or %{python3_sitelib}/* is discouraged in the packaging guidelines, please don't do that -- https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_explicit_lists

(In reply to Miro Hrončok from comment #8)
> 1. The python3-ldns package now lists:
>     %{python3_sitearch}/*
>     %{python3_sitelib}/*
> 
> This means you misunderstood my advice (or decided to deliberately not
> follow it) and packaged some files to sitelib and some files to sitearch:
> 
> /usr/lib/python3.11/site-packages/__pycache__
> /usr/lib/python3.11/site-packages/__pycache__/ldnsx.cpython-311.opt-1.pyc
> /usr/lib/python3.11/site-packages/__pycache__/ldnsx.cpython-311.pyc
> /usr/lib/python3.11/site-packages/ldns
> /usr/lib/python3.11/site-packages/ldnsx.py
> /usr/lib64/python3.11/site-packages/__pycache__
> /usr/lib64/python3.11/site-packages/__pycache__/ldns.cpython-311.opt-1.pyc
> /usr/lib64/python3.11/site-packages/__pycache__/ldns.cpython-311.pyc
> /usr/lib64/python3.11/site-packages/_ldns.so
> /usr/lib64/python3.11/site-packages/_ldns.so.3
> /usr/lib64/python3.11/site-packages/_ldns.so.3.5.0
> /usr/lib64/python3.11/site-packages/ldns.py
> 
> I wonder why.

I did that initially, then reverted it to %python3_sitearch only, like have you recommended. But forgot to remove all remains of sitelib use.
But failed to modify properly Fedora patch to be exactly what upstream has merged.
Original change with both sitelib and sitearch directores were not intended to be used, but were by mistake. Will fix it in following change.
> 
> Anyway, if you do it that way, the files in
> /usr/lib/python3.11/site-packages/__pycache__ now exist in both x86_64 and
> i686 versions of the package and they may or may not be bit-by-bit
> identical. You need to follow
> https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_Appendix/
> #_byte_compilation_reproducibility

Does that make sense when they install only to %python3_sitearch directory and sitelib is not used anymore? Should the compatibility tool used anyway?
> 
> 
> 2. The (nonexistent) python2-ldns package now lists:
>     %{python2_sitearch}/*
>     %{python3_sitelib}/*
> 
> This is probably a typo.
Yes, will remove that anyway.
> 
> 
> 3. Listing %{python3_sitearch}/* and/or %{python3_sitelib}/* is discouraged
> in the packaging guidelines, please don't do that --
> https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/
> #_explicit_lists

Replaced with %{python3_sitearch}/%{name}*, I hope that is okay. Did similar thing also for perl and devel files.

Can you please check the pull request if you find anything else to fix?
https://src.fedoraproject.org/rpms/ldns/pull-request/5

Ouch, pushed update to normal rawhide branch instead of my fork, so building production package anyway.

> Does that make sense when they install only to %python3_sitearch directory and sitelib is not used anymore? Should the compatibility tool used anyway?

No and no :)

> Ouch, pushed update to normal rawhide branch instead of my fork, so building production package anyway.

Looking at the change anyway. It looks correct to me, one nitpick:

This won't work: %pycached %{python2_sitearch}/%{name}.py

See the note in https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/#_byte_compiling "The %pycached macro only supports Python 3.5+"

At this point, I'd consider dropping the python2 bits from the specfile instead of trying to make them compatible with the current Fedora.

This bug appears to have been reported against 'rawhide' during the Fedora Linux 38 development cycle.
Changing version to 38.

I think these fixes are not necessary in previous stable releases unless they are updated too.

Update in F38+:
https://bodhi.fedoraproject.org/updates/FEDORA-2023-6154fb5233