Bug 1627964 - guestfs-python error occur when `import guestfs` on centos 7.5(centos 1804) use python3.
Summary: guestfs-python error occur when `import guestfs` on centos 7.5(centos 1804) u...
Keywords:
Status: NEW
Alias: None
Product: Virtualization Tools
Classification: Community
Component: libguestfs
Version: unspecified
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Richard W.M. Jones
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-09-12 01:27 UTC by wonderkun
Modified: 2020-05-05 21:33 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:


Attachments (Terms of Use)

Description wonderkun 2018-09-12 01:27:15 UTC
Description of problem:

guestfs-python error occur when `import guestfs` on centos 7.5(centos 1804) use python3.  

Version-Release number of selected component (if applicable):

http://download.libguestfs.org/python/guestfs-1.38.4.tar.gz

How reproducible:

[imas@localhost ~]$ scl enable rh-python36 bash
[imas@localhost ~]$ python
Python 3.6.3 (default, Mar 20 2018, 13:50:41)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-16)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import guestfs
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/guestfs.py", line 73, in <module>
    import libguestfsmod
ImportError: /opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/libguestfsmod.cpython-36m-x86_64-linux-gnu.so: undefined symbol: PyString_FromStringAndSize

>>> import libguestfsmod
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: /opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/libguestfsmod.cpython-36m-x86_64-linux-gnu.so: undefined symbol: PyString_FromStringAndSize
>>>


Actual results:


Expected results:


Additional info:

Comment 1 Pino Toscano 2018-09-12 08:09:00 UTC
You need to explain in detail what you did: libguestfs part of the base OS (so python-guestfs is built for Python 2.7), and it is not shipped in that rh-python36 SCL.

Comment 2 Étienne BERSAC 2018-10-11 13:53:36 UTC
Hi,

I confirm the bug with python 3.5 and 3.7.

To reproduce:

1. pip3 install download.libguestfs.org/python/guestfs-1.38.6.tar.gz
2. python -m guestfs

Here is the output:

Traceback (most recent call last):
  File "/home/bersace/.local/share/pyenv/versions/3.7.0/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/bersace/.local/share/pyenv/versions/3.7.0/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/bersace/.local/share/pyenv/versions/3.7.0/envs/aws/lib/python3.7/site-packages/guestfs.py", line 73, in <module>
    import libguestfsmod
ImportError: /home/bersace/.local/share/pyenv/versions/3.7.0/envs/aws/lib/python3.7/site-packages/libguestfsmod.cpython-37m-x86_64-linux-gnu.so: undefined symbol: PyString_FromStringAndSize


guestfs python binding advertise python3 support. Can you point to a recipe to properly build the extension on Python 3 ?

Regards,
Étienne BERSAC

Comment 3 Étienne BERSAC 2018-10-11 14:02:28 UTC
Hi,

I found https://github.com/libguestfs/libguestfs/blob/master/docs/guestfs-building.pod#BUILDING-PYTHON-2-AND-PYTHON-3-BINDINGS . I'm investigating.

Regards,
Étienne BERSAC

Comment 4 Étienne BERSAC 2018-10-11 15:46:45 UTC
As a workaround, you can use GObject Introspection from Python3:

``` python

import gi
gi.require_version('Guestfs', '1.0')
from gi.repository import Guestfs

g = Guestfs.Session()
g.add_drive('guest.img')
g.launch()
g.mount('/dev/sda1', '/')
g.touch('/hello')
g.umount('/')
g.shutdown()
g.close()
```

I don't know if this work on CentOS 7.5.

Regards,

Comment 5 Pino Toscano 2018-10-12 09:05:38 UTC
@Étienne BERSAC:
the Python bindings are definitely not broken, and they work fine when built properly.

I'd still like to know how this is setup in the first place, and have replies to my questions in comment 1.

Comment 6 Pino Toscano 2018-10-12 09:06:39 UTC
Oh, and I still need the same feedback from the reporter, so please do not remove the needinfo for them, thanks.

Comment 7 Radostin Stoyanov 2018-12-22 19:47:40 UTC
I am getting the same error on Arch Linux (where Python 3.7.1 is the default version of Python) when using python-libguestfs from the Arch User Repository:
https://aur.archlinux.org/packages/python-libguestfs/

This is the PKGBUILD file which contains the build recipe:
https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=python-libguestfs

Comment 8 brent s. 2018-12-23 06:57:33 UTC
Hi, I'm the maintainer of the above Arch Linux AUR package.

I can confirm that Arch's system python is python3.7:

[bts@cylon ~]$ export PY=$(which python); echo ${PY}; ls -l ${PY}
/usr/bin/python
lrwxrwxrwx 1 root root 7 Oct 22 06:41 /usr/bin/python -> python3
[bts@cylon ~]$ /usr/bin/python --version
Python 3.7.1


It's out of my control since I don't maintain libguestfs, but the way we build libguestfs is here:

https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=libguestfs

Comment 9 Richard W.M. Jones 2018-12-23 11:10:13 UTC
I believe the problem here is you're trying to use bindings compiled for Python 2
with Python 3 (or maybe vice versa).  This isn't going to work.  What does work
is building libguestfs twice for each version of Python, which is what we do in
Fedora.  It's a PITA, but you can read the Fedora spec to see how to do it:

https://src.fedoraproject.org/rpms/libguestfs/blob/master/f/libguestfs.spec#_863

Comment 10 brent s. 2018-12-23 11:19:25 UTC
Thanks - I'll pass this along to the libguestfs maintainer. Since we (Arch) are on python3 as the default though (and given python2's EOL for 1.1.2020), I think we can get away with just doing python3 bindings instead of compiling both.

Comment 11 Sam 2019-01-13 10:09:17 UTC
Hey, this bug is in not in libguestfs per-se but in the standalone python package:

http://download.libguestfs.org/python/guestfs-1.38.2.tar.gz (for example)

The file config.h was created somewhere else (on the machine of the user that created this package?):

/* config.h.  Generated from config.h.in by configure.  */⏎
/* config.h.in.  Generated from configure.ac by autoheader.  */⏎

And it was with python2 - so the entire file is misconfigured.

#define HAVE_PYSTRING_ASSTRING 1 - is just one symptom of it - reflected in handle.c.

This package should use the python definition which are available from the current python build and keep config.h clean of python configuration.

Sam

Comment 12 Richard W.M. Jones 2019-01-14 12:30:01 UTC
Should do that indeed, but in fact it's quite hard to do it practically.
I believe only the HAVE_PYSTRING_ASSTRING setting needs to be changed though.

Comment 13 Sam 2019-01-14 12:37:44 UTC
(In reply to Richard W.M. Jones from comment #12)
> Should do that indeed, but in fact it's quite hard to do it practically.
> I believe only the HAVE_PYSTRING_ASSTRING setting needs to be changed though.

Can we just create 2 sdists - one for python2 one for python3 in the meanwhile?

The alternatives are:
 1. Edit the file manually after downloading the package - which is not that "nice" (it will require to manually extract the files and run setup.py).
 2. Install tons of packages on my machine to compile libguestfs manually with PYTHON=python36 - this is actually pretty hard since there are a lot of dependancies and it took me a while to get it right (--with-distro=REDHAT on oraclelinux was an elusive parameter)

Sam

Comment 14 Pino Toscano 2019-01-14 12:47:22 UTC
(In reply to Sam from comment #13)
> (--with-distro=REDHAT on oraclelinux was an elusive parameter)

Feel free to propose a small addition to m4/guestfs-appliance.m4.

Comment 15 Richard W.M. Jones 2019-01-17 14:16:24 UTC
Starting with 1.40 the Python packages will be built for Python 3.

http://download.libguestfs.org/python/

Comment 16 Sam 2019-01-17 14:18:48 UTC
(In reply to Richard W.M. Jones from comment #15)
> Starting with 1.40 the Python packages will be built for Python 3.
> 
> http://download.libguestfs.org/python/

That's great!
Thanks

Comment 17 Sam 2019-01-19 08:50:08 UTC
Regarding the implementation of the support for python3.

At the moment m4/guestfs-python.m4 contains the following defines:

AC_DEFINE([HAVE_PYTHON],...
AC_DEFINE([HAVE_PYCAPSULE_NEW],...
AC_DEFINE([HAVE_PYSTRING_ASSTRING],..

The HAVE_PYTHON define does not hold much importance with regard to py2/py3.

I believe HAVE_PYCAPSULE_NEW and HAVE_PYSTRING_ASSTRING should be removed from m4/guestfs-python.m4, the check whether the symbol is contained in libpython is not the correct way of creating cross version python extensions.

I think HAVE_PYCAPSULE_NEW should be defined in generator/python.ml as:

#if (PY_VERSION_HEX >= 0x02070000 && PY_VERSION_HEX < 0x03000000) || PY_VERSION_HEX >= 0x03010000
     #define HAVE_PYCAPSULE_NEW
#endif

I think HAVE_PYSTRING_ASSTRING should be defined in python/handle.c as:

#if PY_MAJOR_VERSION <= 2
     #define HAVE_PYSTRING_ASSTRING
#endif

This way we can have one source distribution for py2/py3

Sam

Comment 18 Sam 2019-01-28 17:59:26 UTC
It's me again,

So after successfully compiling libguestfs bindings for python3 (manually) I am here to report errors in the same location.

At the moment the bindings are broken with regard to utf-8. (Not trying to sound dramatic :-) )

When I call "g.read_file(path)" on a binary file I get a utf-8 error.

After investigating shortly - makes sense:

In python2 we called (handle.c):

PyString_FromStringAndSize - this works great since it works for bytes & strings in Python2

In python3 we now call:

PyUnicode_FromStringAndSize - this does not work well on binary files.
In face, we should call PyBytes_FromStringAndSize.

So what I did is I changed:
PyUnicode_FromString -> PyBytes_FromString
PyUnicode_FromStringAndSize -> PyBytes_FromStringAndSize
While not changing (guestfs_int_py_asstring)

But now functions that are supposed to return utf-8 string objects suchs as g.get_cachedir() return the path as a bytes object - which is not logical.

So my final fix was to just change:

PyUnicode_FromStringAndSize -> PyBytes_FromStringAndSize

This currently works since g.get_cachedir() uses PyUnicode_FromString and g.read_file(path) uses PyBytes_FromStringAndSize.
But maybe its just luck and there are functions which will be mixed (functions that should return strings but will use PyBytes_FromStringAndSize now or vice-versa)

This is something that should be taken care of in the framework that generates the "actions-i.c" files.

Sam

Comment 19 Pino Toscano 2019-01-29 10:07:47 UTC
(In reply to Sam from comment #18)
> It's me again,
> 
> So after successfully compiling libguestfs bindings for python3 (manually) I
> am here to report errors in the same location.
> 
> At the moment the bindings are broken with regard to utf-8. (Not trying to
> sound dramatic :-) )
> 
> When I call "g.read_file(path)" on a binary file I get a utf-8 error.
> [...]

This is a different issue: bug 1661871.

Comment 20 Sam 2019-01-29 10:23:06 UTC
(In reply to Pino Toscano from comment #19)
> (In reply to Sam from comment #18)
> > It's me again,
> > 
> > So after successfully compiling libguestfs bindings for python3 (manually) I
> > am here to report errors in the same location.
> > 
> > At the moment the bindings are broken with regard to utf-8. (Not trying to
> > sound dramatic :-) )
> > 
> > When I call "g.read_file(path)" on a binary file I get a utf-8 error.
> > [...]
> 
> This is a different issue: bug 1661871.

Missed it.

Although the issue is very related (a single fix can solve both of the issues).

My current workaround (as I specified above) is:

sed -i 's/PyUnicode_FromStringAndSize/PyBytes_FromStringAndSize/g' $LIBGUESTFS_DIR/python/handle.c


Note You need to log in before you can comment on or make changes to this bug.