Bug 1627964

You need to explain in detail what you did: libguestfs part of the base OS (so python-guestfs is built for Python 2.7), and it is not shipped in that rh-python36 SCL.

Hi,

I confirm the bug with python 3.5 and 3.7.

To reproduce:

1. pip3 install download.libguestfs.org/python/guestfs-1.38.6.tar.gz
2. python -m guestfs

Here is the output:

Traceback (most recent call last):
  File "/home/bersace/.local/share/pyenv/versions/3.7.0/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/bersace/.local/share/pyenv/versions/3.7.0/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/bersace/.local/share/pyenv/versions/3.7.0/envs/aws/lib/python3.7/site-packages/guestfs.py", line 73, in <module>
    import libguestfsmod
ImportError: /home/bersace/.local/share/pyenv/versions/3.7.0/envs/aws/lib/python3.7/site-packages/libguestfsmod.cpython-37m-x86_64-linux-gnu.so: undefined symbol: PyString_FromStringAndSize


guestfs python binding advertise python3 support. Can you point to a recipe to properly build the extension on Python 3 ?

Regards,
Étienne BERSAC

Hi,

I found https://github.com/libguestfs/libguestfs/blob/master/docs/guestfs-building.pod#BUILDING-PYTHON-2-AND-PYTHON-3-BINDINGS . I'm investigating.

Regards,
Étienne BERSAC

As a workaround, you can use GObject Introspection from Python3:

``` python

import gi
gi.require_version('Guestfs', '1.0')
from gi.repository import Guestfs

g = Guestfs.Session()
g.add_drive('guest.img')
g.launch()
g.mount('/dev/sda1', '/')
g.touch('/hello')
g.umount('/')
g.shutdown()
g.close()
```

I don't know if this work on CentOS 7.5.

Regards,

@Étienne BERSAC:
the Python bindings are definitely not broken, and they work fine when built properly.

I'd still like to know how this is setup in the first place, and have replies to my questions in comment 1.

Oh, and I still need the same feedback from the reporter, so please do not remove the needinfo for them, thanks.

I am getting the same error on Arch Linux (where Python 3.7.1 is the default version of Python) when using python-libguestfs from the Arch User Repository:
https://aur.archlinux.org/packages/python-libguestfs/

This is the PKGBUILD file which contains the build recipe:
https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=python-libguestfs

Hi, I'm the maintainer of the above Arch Linux AUR package.

I can confirm that Arch's system python is python3.7:

[bts@cylon ~]$ export PY=$(which python); echo ${PY}; ls -l ${PY}
/usr/bin/python
lrwxrwxrwx 1 root root 7 Oct 22 06:41 /usr/bin/python -> python3
[bts@cylon ~]$ /usr/bin/python --version
Python 3.7.1


It's out of my control since I don't maintain libguestfs, but the way we build libguestfs is here:

https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=libguestfs

I believe the problem here is you're trying to use bindings compiled for Python 2
with Python 3 (or maybe vice versa).  This isn't going to work.  What does work
is building libguestfs twice for each version of Python, which is what we do in
Fedora.  It's a PITA, but you can read the Fedora spec to see how to do it:

https://src.fedoraproject.org/rpms/libguestfs/blob/master/f/libguestfs.spec#_863

Thanks - I'll pass this along to the libguestfs maintainer. Since we (Arch) are on python3 as the default though (and given python2's EOL for 1.1.2020), I think we can get away with just doing python3 bindings instead of compiling both.

Hey, this bug is in not in libguestfs per-se but in the standalone python package:

http://download.libguestfs.org/python/guestfs-1.38.2.tar.gz (for example)

The file config.h was created somewhere else (on the machine of the user that created this package?):

/* config.h.  Generated from config.h.in by configure.  */⏎
/* config.h.in.  Generated from configure.ac by autoheader.  */⏎

And it was with python2 - so the entire file is misconfigured.

#define HAVE_PYSTRING_ASSTRING 1 - is just one symptom of it - reflected in handle.c.

This package should use the python definition which are available from the current python build and keep config.h clean of python configuration.

Sam

Should do that indeed, but in fact it's quite hard to do it practically.
I believe only the HAVE_PYSTRING_ASSTRING setting needs to be changed though.

(In reply to Richard W.M. Jones from comment #12)
> Should do that indeed, but in fact it's quite hard to do it practically.
> I believe only the HAVE_PYSTRING_ASSTRING setting needs to be changed though.

Can we just create 2 sdists - one for python2 one for python3 in the meanwhile?

The alternatives are:
 1. Edit the file manually after downloading the package - which is not that "nice" (it will require to manually extract the files and run setup.py).
 2. Install tons of packages on my machine to compile libguestfs manually with PYTHON=python36 - this is actually pretty hard since there are a lot of dependancies and it took me a while to get it right (--with-distro=REDHAT on oraclelinux was an elusive parameter)

Sam

(In reply to Sam from comment #13)
> (--with-distro=REDHAT on oraclelinux was an elusive parameter)

Feel free to propose a small addition to m4/guestfs-appliance.m4.

Starting with 1.40 the Python packages will be built for Python 3.

http://download.libguestfs.org/python/

(In reply to Richard W.M. Jones from comment #15)
> Starting with 1.40 the Python packages will be built for Python 3.
> 
> http://download.libguestfs.org/python/

That's great!
Thanks

Regarding the implementation of the support for python3.

At the moment m4/guestfs-python.m4 contains the following defines:

AC_DEFINE([HAVE_PYTHON],...
AC_DEFINE([HAVE_PYCAPSULE_NEW],...
AC_DEFINE([HAVE_PYSTRING_ASSTRING],..

The HAVE_PYTHON define does not hold much importance with regard to py2/py3.

I believe HAVE_PYCAPSULE_NEW and HAVE_PYSTRING_ASSTRING should be removed from m4/guestfs-python.m4, the check whether the symbol is contained in libpython is not the correct way of creating cross version python extensions.

I think HAVE_PYCAPSULE_NEW should be defined in generator/python.ml as:

#if (PY_VERSION_HEX >= 0x02070000 && PY_VERSION_HEX < 0x03000000) || PY_VERSION_HEX >= 0x03010000
     #define HAVE_PYCAPSULE_NEW
#endif

I think HAVE_PYSTRING_ASSTRING should be defined in python/handle.c as:

#if PY_MAJOR_VERSION <= 2
     #define HAVE_PYSTRING_ASSTRING
#endif

This way we can have one source distribution for py2/py3

Sam

It's me again,

So after successfully compiling libguestfs bindings for python3 (manually) I am here to report errors in the same location.

At the moment the bindings are broken with regard to utf-8. (Not trying to sound dramatic :-) )

When I call "g.read_file(path)" on a binary file I get a utf-8 error.

After investigating shortly - makes sense:

In python2 we called (handle.c):

PyString_FromStringAndSize - this works great since it works for bytes & strings in Python2

In python3 we now call:

PyUnicode_FromStringAndSize - this does not work well on binary files.
In face, we should call PyBytes_FromStringAndSize.

So what I did is I changed:
PyUnicode_FromString -> PyBytes_FromString
PyUnicode_FromStringAndSize -> PyBytes_FromStringAndSize
While not changing (guestfs_int_py_asstring)

But now functions that are supposed to return utf-8 string objects suchs as g.get_cachedir() return the path as a bytes object - which is not logical.

So my final fix was to just change:

PyUnicode_FromStringAndSize -> PyBytes_FromStringAndSize

This currently works since g.get_cachedir() uses PyUnicode_FromString and g.read_file(path) uses PyBytes_FromStringAndSize.
But maybe its just luck and there are functions which will be mixed (functions that should return strings but will use PyBytes_FromStringAndSize now or vice-versa)

This is something that should be taken care of in the framework that generates the "actions-i.c" files.

Sam

(In reply to Sam from comment #18)
> It's me again,
> 
> So after successfully compiling libguestfs bindings for python3 (manually) I
> am here to report errors in the same location.
> 
> At the moment the bindings are broken with regard to utf-8. (Not trying to
> sound dramatic :-) )
> 
> When I call "g.read_file(path)" on a binary file I get a utf-8 error.
> [...]

This is a different issue: bug 1661871.

(In reply to Pino Toscano from comment #19)
> (In reply to Sam from comment #18)
> > It's me again,
> > 
> > So after successfully compiling libguestfs bindings for python3 (manually) I
> > am here to report errors in the same location.
> > 
> > At the moment the bindings are broken with regard to utf-8. (Not trying to
> > sound dramatic :-) )
> > 
> > When I call "g.read_file(path)" on a binary file I get a utf-8 error.
> > [...]
> 
> This is a different issue: bug 1661871.

Missed it.

Although the issue is very related (a single fix can solve both of the issues).

My current workaround (as I specified above) is:

sed -i 's/PyUnicode_FromStringAndSize/PyBytes_FromStringAndSize/g' $LIBGUESTFS_DIR/python/handle.c