Bug 1076293 - distutils.sysconf.get_python_lib() API broken vs. prefix by lib64 patch
Summary: distutils.sysconf.get_python_lib() API broken vs. prefix by lib64 patch
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: python
Version: 20
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Matej Stuchlik
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-03-14 03:16 UTC by Allison Karlitskaya
Modified: 2016-02-01 02:15 UTC (History)
10 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2015-06-29 19:27:21 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Allison Karlitskaya 2014-03-14 03:16:29 UTC
The docs say this:

get_python_lib(plat_specific=0, standard_lib=0, prefix=None)
    Return the directory containing the Python library (standard or
    site additions).
    
    If 'plat_specific' is true, return the directory containing
    platform-specific modules, i.e. any module from a non-pure-Python
    module distribution; otherwise, return the platform-shared library
    directory.  If 'standard_lib' is true, return the directory
    containing standard Python library modules; otherwise, return the
    directory for site-specific modules.
    
    If 'prefix' is supplied, use it instead of sys.prefix or
    sys.exec_prefix -- i.e., ignore 'plat_specific'.


In particular, note the last paragraph: plat_specific should be ignored if prefix is suppied.

python-2.7.3-lib64.patch breaks this feature of the API.

After the patch, the code reads like so:

    if prefix is None:
        prefix = plat_specific and EXEC_PREFIX or PREFIX

    if os.name == "posix":
        if plat_specific or standard_lib:
            lib = "lib64"
        else:
            lib = "lib"
        libpython = os.path.join(prefix,
                                 lib, "python" + get_python_version())
        if standard_lib:
            return libpython
        else:
            return os.path.join(libpython, "site-packages")

with the 'if plat_specific or standard_lib' lib64 part inserted by the Fedora patch.

This looks to have happened as part of bug 125472.

I think the correct fix here is to also add a 'and not prefix' to that conditional (ie: lib64 patch only applies to the system directories).

This currently impacts jhbuild GNOME which installs 3rd party python modules into a custom prefix.  In particular, we assume that (as per the docs) this function will only return a single site-packages directory if we give a prefix= kwarg.

It's also questionable if 'standard_lib' should be consulted at all here... but we don't even attempt to install anything there, so it doesn't really impact us.

Comment 1 Toshio Ernie Kuratomi 2014-03-14 08:38:46 UTC
Had a look at the code and I could see this being evaluated either way.
It looks like that patch adds additional meaning to what setting plat_specific
means.  The documentation does not make the same sort of natural sense for the
new meaning as it does for the original meaning.

The upstream code uses plat_specific to determine whether the prefix should be
set to PREFIX or to EXECPREFIX (as passed in to configure during the build of
the python interpreter).  In this context, the documentation makes perfect
sense:

"If 'prefix' is supplied, use it instead of sys.prefix or sys.exec_prefix" is
stating that a parameter given as prefix is overriding the PREFIX and
EXECPREFIX choices that python has stored from running configure.

"i.e., ignore 'plat_specific" is giving the natural expansion of that override to what is implemented in the upstream code.  The one purpose of plat_specific there is being overridden by the explicit prefix setting.

What this patch does is make setting plat_specific have a second purpose.  In
addition to controlling the prefix, it also controls the choice of
subdirectory within the prefix.  This second meaning has a strange
relationship to the documentation.  The portion of the documentation that
talks about the meaning of prefix does not apply to this because the prefix
does not come into play here.  However, the expansion that attempts to clarify
what the end result of setting prefix no longer follows from the new
meaning.  Overriding the remembered, default prefix does not have any natural
effect on the subdirectory within the prefix that is chosen.  So probably it is
the documentation that is wrong here rather than the code.  I'd suggest
patching the documentation to say something like "i.e. use the directory
specified in prefix instead of the one that plat_specific would have pulled
from python's compile-time values"

However, interpreting it that way wouldn't be without controversy.  Even
though the structure of the sentences in the documentation show that
ignoring plat_specific is just the natural consequence of setting an explicit
prefix one could decide that the documentation is a contract that the
code must follow (I'd note that upstream python doesn't see things this way.
They sometimes adjust the documentation when they find it doesn't agree with
the intentions of the code).  If this view is taken then the patch should
indeed ignore plat_specific if prefix is supplied.  One thing to note
then is that the subdirectory needs to be set to lib64 (on multilib systems)
rather than lib in this case.  This is because it is okay to place arch
independent content into an arch specific location but it isn't okay to place
arch specific content into an arch independent location.

So if this interpretation is followed, the patch would need to look something like this:

     If 'prefix' is supplied, use it instead of sys.prefix or
     sys.exec_prefix -- i.e., ignore 'plat_specific'.
     """
+    ignore_plat_specific = True
     if prefix is None:
+        ignore_plat_specific = False
         prefix = plat_specific and EXEC_PREFIX or PREFIX
 
     if os.name == "posix":
+        if ignore_plat_specific or standard_lib or plat_specific:
+            lib = "lib64"
+        else:
+            lib = "lib"
         libpython = os.path.join(prefix,
-                                 "lib", "python" + get_python_version())
+                                 lib, "python" + get_python_version())
         if standard_lib:
             return libpython
         else:

Comment 2 Allison Karlitskaya 2014-03-14 12:20:53 UTC
I disagree with this logic.  jhbuild is building upstream packages right out of git, with no special patches.  It just wants to use normal automake-style behaviour here (ie: libdir is ${prefix}/lib).  We don't do multiarch inside the jhbuild dierctory so we have no such concern.

Multiarch is something that Fedora has decided to do in its /usr.  It should not impact people doing things in other locations.

fwiw, I think it *may* be desirable to check if prefix is explicitly given as '/usr' and enable the lib/lib64 distinction.  The source of this problem is an automake check that calls this function with prefix= set from the --prefix given to the automake script.  If you call it with --prefix=/usr then I guess you'd probably expect the Fedora-specific behaviour to kick in for that case...

Comment 3 Toshio Ernie Kuratomi 2014-03-14 15:30:41 UTC
That sounds like setting libdir = ${prefix}/lib is what you should be doing.  It's not the value of prefix (in the configure sense) that's being overridden here, it's the value of libdir.  Since you're using the distutils.sysconfig functions, though, I'm not certain that setting the configure option is sufficient.

Comment 4 Allison Karlitskaya 2014-03-14 17:07:10 UTC
automake defines pyexecdir like so:

    exec_prefix=$prefix

...

  PYTHON_PREFIX='${prefix}'

  PYTHON_EXEC_PREFIX='${exec_prefix}'

...


  am_cv_python_pyexecdir=`$PYTHON -c "from distutils import sysconfig; print(sysconfig.get_python_lib(1,0,prefix='$PYTHON_EXEC_PREFIX'))" 2>/dev/null ||
     echo "${PYTHON_EXEC_PREFIX}/lib/python${PYTHON_VERSION}/site-packages"`


ie: it tries to use get_python_lib() to get the correctly-versioned path from Python itself.  Failing that, it falls back to guessing it, hardcoded to ${prefix}/lib/pythonN.N/site-packages.  libdir doesn't come into play at all here.

Even if it did, the default value of libdir is ${prefix}/lib, so setting --libdir=${prefix}/lib would essentially be a no-op.

Comment 5 Fedora Admin XMLRPC Client 2015-05-12 12:03:03 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 6 Fedora End Of Life 2015-05-29 11:14:52 UTC
This message is a reminder that Fedora 20 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 20. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '20'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 20 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 7 Fedora End Of Life 2015-06-29 19:27:21 UTC
Fedora 20 changed to end-of-life (EOL) status on 2015-06-23. Fedora 20 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.