Bug 817554 - UsrMove broke Py_Initialize()
UsrMove broke Py_Initialize()
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: python (Show other bugs)
17
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: Dave Malcolm
Fedora Extras Quality Assurance
:
Depends On:
Blocks: 903985
  Show dependency treegraph
 
Reported: 2012-04-30 09:32 EDT by Jan Safranek
Modified: 2013-03-28 10:59 EDT (History)
6 users (show)

See Also:
Fixed In Version: python-2.7.3-34.fc19
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 903985 (view as bug list)
Environment:
Last Closed: 2013-03-06 17:34:13 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
reproducer (87 bytes, text/plain)
2012-04-30 09:32 EDT, Jan Safranek
no flags Details

  None (edit)
Description Jan Safranek 2012-04-30 09:32:38 EDT
Created attachment 581187 [details]
reproducer

Description of problem:
I have a daemon with embedded python. When the daemon is started via systemd (->working directory is /), following code fails:

  Py_SetProgramName("cmpi_swig");
  Py_Initialize();

The code is not mine, it's from cmpi-bindings project and was working perfectly in older Fedora.

Version-Release number of selected component (if applicable):
python-2.7.3-4.fc18.x86_64

How reproducible:
always

Steps to Reproduce:
1. compile attached reproducer: gcc -o reproduce reproduce.c -lpython2.7 -I/usr/include/python2.7
2. cd /
3. $OLDPWD/reproduce
  
Actual results:
Traceback (most recent call last):
  File "/lib64/python2.7/site.py", line 567, in <module>
    main()
  File "/lib64/python2.7/site.py", line 549, in main
    known_paths = addusersitepackages(known_paths)
  File "/lib64/python2.7/site.py", line 278, in addusersitepackages
    user_site = getusersitepackages()
  File "/lib64/python2.7/site.py", line 253, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "/lib64/python2.7/site.py", line 243, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/lib64/python2.7/sysconfig.py", line 521, in get_config_var
    return get_config_vars().get(name)
  File "/lib64/python2.7/sysconfig.py", line 420, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/lib64/python2.7/sysconfig.py", line 299, in _init_posix
    raise IOError(msg)
IOError: invalid Python installation: unable to open //include/python2.7/pyconfig-64.h (No such file or directory)


Expected results:
no error reported

Additional info:
Looking at strace output, python tries to 'find itself'. On Fedora16 (=before usrmove):
stat("/usr/lib64/qt-3.3/bin/cmpi_swig", 0x7fffc439d0e0) = -1 ENOENT (No such file or directory)
stat("/usr/local/bin/cmpi_swig", 0x7fffc439d0e0) = -1 ENOENT (No such file or directory)
stat("/usr/bin/cmpi_swig", 0x7fffc439d0e0) = -1 ENOENT (No such file or directory)
stat("/bin/cmpi_swig", 0x7fffc439d0e0)  = -1 ENOENT (No such file or directory)
stat("/usr/games/cmpi_swig", 0x7fffc439d0e0) = -1 ENOENT (No such file or directory)
stat("/usr/local/sbin/cmpi_swig", 0x7fffc439d0e0) = -1 ENOENT (No such file or directory)
stat("/usr/sbin/cmpi_swig", 0x7fffc439d0e0) = -1 ENOENT (No such file or directory)
stat("/sbin/cmpi_swig", 0x7fffc439d0e0) = -1 ENOENT (No such file or directory)
stat("/home/jsafrane/bin/cmpi_swig", 0x7fffc439d0e0) = -1 ENOENT (No such file or directory)
stat("/opt/eclipse/cmpi_swig", 0x7fffc439d0e0) = -1 ENOENT (No such file or directory)
readlink("", 0x7fffc439f190, 4096)      = -1 ENOENT (No such file or directory)
stat("Modules/Setup", 0x7fffc439d030)   = -1 ENOENT (No such file or directory)
getcwd("/", 4096)                       = 2
stat("/lib64/python2.7/os.py", 0x7fffc439d020) = -1 ENOENT (No such file or directory)
stat("/lib64/python2.7/os.pyc", 0x7fffc439d020) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/python2.7/os.py", {st_mode=S_IFREG|0644, st_size=26300, ...}) = 0
stat("Modules/Setup", 0x7fffc439d030)   = -1 ENOENT (No such file or directory)
getcwd("/", 4096)                       = 2
stat("/lib64/python2.7/lib-dynload", 0x7fffc439d030) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/python2.7/lib-dynload", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0

-> it *does not* find itself in / (current working directory) and continues with /usr/

On Rawhide:
stat("/usr/local/sbin/cmpi_swig", 0x7fffd54784f0) = -1 ENOENT (No such file or directory)
stat("/sbin/cmpi_swig", 0x7fffd54784f0) = -1 ENOENT (No such file or directory)
stat("/bin/cmpi_swig", 0x7fffd54784f0)  = -1 ENOENT (No such file or directory)
stat("/usr/sbin/cmpi_swig", 0x7fffd54784f0) = -1 ENOENT (No such file or directory)
stat("/usr/bin/cmpi_swig", 0x7fffd54784f0) = -1 ENOENT (No such file or directory)
stat("/root/bin/cmpi_swig", 0x7fffd54784f0) = -1 ENOENT (No such file or directory)
readlink("", 0x7fffd5479590, 4096)      = -1 ENOENT (No such file or directory)
stat("Modules/Setup", 0x7fffd5478440)   = -1 ENOENT (No such file or directory)
getcwd("/", 4096)                       = 2
stat("/lib64/python2.7/os.py", {st_mode=S_IFREG|0644, st_size=25769, ...}) = 0
stat("Modules/Setup", 0x7fffd5478440)   = -1 ENOENT (No such file or directory)
getcwd("/", 4096)                       = 2
stat("/lib64/python2.7/lib-dynload", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0

-> it *finds* itself in / (/lib64/python2.7/lib-dynload) and expect //include/python2.7/pyconfig-64.h exists -> error
Comment 1 Dave Malcolm 2012-05-02 15:52:08 EDT
Thanks for filing this bug report.  I'm able to reproduce this, and am investigating.

Note to self:
This is happening because sys.path is erroneously getting initialized to:
   ['lib64/python27.zip', '/lib64/python2.7/', '/lib64/python2.7/plat-linux2', '/lib64/python2.7/lib-tk', '/lib64/python2.7/lib-old', '/lib64/python2.7/lib-dynload']
rather than to the "/usr/"-prefixed paths.

The initial value for sys.path comes from calculate_path(), in Modules/getpath.c

Summary of the logic in calculate_path():
  if there's a slash in Py_GetProgramName()
  then:
    it uses Py_GetProgramName() as argv0_path; get the containing dir as argv0_path
  else:
    it tries to find Py_GetProgramName() within $PATH
    if found:
       it uses the abspath it finds; get the containing dir as argv0_path
    else: (not found on $PATH):
       argv0_path == ""
  if PYTHONHOME not set (step 1):
    if argv0_path is not in the build directory (step 2): 
       Step 3: search for "/lib64/python2.7/os.py" in argv0_path:
         (*) the bug is happening due to it finds this; previously it wouldn't have
       Step 4: search within PREFIX/EXEC_PREFIX from preprocessor (which are set to /usr at build time by the Makefile)
         (**) previously it was using this

So it appears that this bug will happen when:
  (1) there is no slash in Py_GetProgramName() OR it is a path to within the root directory (defaults to "python")
  AND 
  (2) Py_GetProgramName() is *not* found in $PATH (defaults to "python", so this *is* normally found, albeit as/bin/python, but since /bin/lib64 doesn't exist the test at step 3 fails)
  AND
  (3) $PYTHONHOME is not set

It looks like previously this combination would have fallen back to "Step 4", using the PREFIX and EXEC_PREFIX from the Makefile, but now it's getting a false success at "Step 3".
Comment 2 Jan Safranek 2012-09-06 04:27:44 EDT
Any progress with this bug? I am still able to reproduce it with python-2.7.3-13.fc18.x86_64 on Fedora 18 pre-alpha.
Comment 3 SpuyMore 2013-01-02 15:23:46 EST
Same for me, I have to symlink /include to /usr/include to make the Python config work.
Comment 4 Dave Malcolm 2013-03-06 15:05:16 EST
Candidate workaround committed to git as:
http://pkgs.fedoraproject.org/cgit/python.git/commit/?id=4e8ad3c0992c75b72400b3a99c7886d275142ebc

Building python-2.7.3-33.fc19 for rawhide
Task info: http://koji.fedoraproject.org/koji/taskinfo?taskID=5086641
Comment 5 Dave Malcolm 2013-03-06 16:40:35 EST
(In reply to comment #4)
> Candidate workaround committed to git as:
> http://pkgs.fedoraproject.org/cgit/python.git/commit/
> ?id=4e8ad3c0992c75b72400b3a99c7886d275142ebc
> 
> Building python-2.7.3-33.fc19 for rawhide
> Task info: http://koji.fedoraproject.org/koji/taskinfo?taskID=5086641
Failed test_sys and test_subprocess when run from the build directory.


Revised candidate workaround committed to git as:
http://pkgs.fedoraproject.org/cgit/python.git/commit/?id=1b2a274a94e296f929715acd189bbe6c028861b1

Building python-2.7.3-34.fc19 for rawhide
Task info: http://koji.fedoraproject.org/koji/taskinfo?taskID=5086849
Comment 6 Dave Malcolm 2013-03-06 17:34:13 EST
Build of python-2.7.3-34.fc19 succeeded
Comment 7 SpuyMore 2013-03-28 10:59:17 EDT
Is this going to be fixed for fc18 as well?

Note You need to log in before you can comment on or make changes to this bug.