Bug 1711638 - samba fails to build with Python 3.8.0a4 (waf compatibility problem)
Summary: samba fails to build with Python 3.8.0a4 (waf compatibility problem)
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: samba
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
Assignee: Guenther Deschner
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1712602 (view as bug list)
Depends On:
Blocks: PYTHON38 1712602
TreeView+ depends on / blocked
 
Reported: 2019-05-19 07:43 UTC by Miro Hrončok
Modified: 2019-07-01 18:05 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-07-01 18:05:04 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
Full log from Copr (264.15 KB, text/plain)
2019-05-19 07:43 UTC, Miro Hrončok
no flags Details
Patch for all libtdb, libtalloc, libtevent, libldb and samba (970 bytes, patch)
2019-05-24 09:55 UTC, Miro Hrončok
no flags Details | Diff


Links
System ID Private Priority Status Summary Last Updated
Gitlab ita1024/waf/issues/2239 0 None None None 2019-05-22 13:57:54 UTC
Python 36721 0 None None None 2019-05-22 09:18:35 UTC
Samba Project 13960 0 None None None 2019-07-01 10:15:18 UTC

Description Miro Hrončok 2019-05-19 07:43:24 UTC
Created attachment 1570816 [details]
Full log from Copr

libtdb-1.3.18-3.fc31 fails to build with 3.8.0a4 in https://copr.fedorainfracloud.org/coprs/g/python/python3.8/

Testing pyembed configuration                                                                   : Could not build a python embedded interpreter 
Testing pyembed configuration                                                                   : Could not build a python embedded interpreter 
...
Traceback (most recent call last):
  File "python/tests/simple.py", line 14, in <module>
    import tdb
ImportError: No module named tdb
Traceback (most recent call last):
  File "python/tests/simple.py", line 14, in <module>
    import tdb
ModuleNotFoundError: No module named 'tdb'
...
Running Python test with /usr/bin/python2: python/tests/simple.py
Python test failed: PYTHONPATH=bin/python /usr/bin/python2 python/tests/simple.py
Running Python test with /usr/bin/python3: python/tests/simple.py
Python test failed: PYTHONPATH=bin/python /usr/bin/python3 python/tests/simple.py
python testsuite returned 1
make: *** [Makefile:17: test] Error 1
error: Bad exit status from /var/tmp/rpm-tmp.ToUNLv (%check)

Full log attached. See bz1710767 for a similar problem (fixed) in python-wxpython4.

Comment 1 Lukas Slebodnik 2019-05-20 08:05:35 UTC
There are dependencies between libraries provided by samba

libtdb -> libldb
libtalloc -> libtevent -> libldb

And I cannot see good build for libtdb
https://copr.fedorainfracloud.org/coprs/g/python/python3.8/build/911193/

Comment 2 Miro Hrončok 2019-05-20 08:43:28 UTC
You cannot see a good build of libtdb, because this is a bug for libtdb failing to build.

Comment 3 Lukas Slebodnik 2019-05-20 09:25:13 UTC
(In reply to Miro Hrončok from comment #2)
> You cannot see a good build of libtdb, because this is a bug for libtdb
> failing to build.

Sorry, I misread libldb/libtdb
Will try do debug in mock later today.

Comment 4 Miro Hrončok 2019-05-20 09:43:08 UTC
> Sorry, I misread libldb/libtdb

Happens to me all the time.

Comment 5 Lukas Slebodnik 2019-05-20 10:38:14 UTC
I would say there is some isseu with python3-config

It does not return -lpython3.8 among linker flags

<mock-chroot> sh-5.0# /usr/bin/python3-config --cflags --libs --ldflags
-I/usr/include/python3.8 -I/usr/include/python3.8  -Wno-unused-result -Wsign-compare  -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches   -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv  -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG  -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches   -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv
 -lcrypt -lpthread -ldl  -lutil -lm -lm 
 -L/usr/lib64  -lcrypt -lpthread -ldl  -lutil -lm -lm 

python2-config works well

<mock-chroot> sh-5.0# /usr/bin/python2-config --cflags --libs --ldflags
-I/usr/include/python2.7 -I/usr/include/python2.7 -fno-strict-aliasing -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv
-lpython2.7 -lpthread -ldl -lutil -lm
-lpython2.7 -lpthread -ldl -lutil -lm -Xlinker -export-dynamic

Comment 6 Lukas Slebodnik 2019-05-20 10:38:30 UTC
<mock-chroot> sh-5.0# /usr/bin/python3-config -h
Usage: /usr/bin/python3.8-x86_64-config --prefix|--exec-prefix|--includes|--libs|--cflags|--ldflags|--extension-suffix|--help|--abiflags|--configdir
<mock-chroot> sh-5.0# /usr/bin/python2-config
Usage: /usr/bin/python2-config [--prefix|--exec-prefix|--includes|--libs|--cflags|--ldflags|--extension-suffix|--help]
<mock-chroot> sh-5.0# /usr/bin/python2-config --extension-suffix
.so
<mock-chroot> sh-5.0# /usr/bin/python3-config --extension-suffix
.cpython-38-x86_64-linux-gnu.so
<mock-chroot> sh-5.0# /usr/bin/python3-config --prefix
/usr
<mock-chroot> sh-5.0# /usr/bin/python3-config --exec-prefix
/usr
<mock-chroot> sh-5.0# /usr/bin/python3-config --includes
-I/usr/include/python3.8 -I/usr/include/python3.8
<mock-chroot> sh-5.0# /usr/bin/python3-config --libs
 -lcrypt -lpthread -ldl  -lutil -lm -lm 
<mock-chroot> sh-5.0# /usr/bin/python3-config --cflags
-I/usr/include/python3.8 -I/usr/include/python3.8  -Wno-unused-result -Wsign-compare  -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches   -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv  -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG  -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches   -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv
<mock-chroot> sh-5.0# /usr/bin/python3-config --ldflags
 -L/usr/lib64  -lcrypt -lpthread -ldl  -lutil -lm -lm 
<mock-chroot> sh-5.0# /usr/bin/python3-config --extension-suffix
.cpython-38-x86_64-linux-gnu.so
<mock-chroot> sh-5.0# /usr/bin/python3-config --abiflags

<mock-chroot> sh-5.0# /usr/bin/python3-config --configdir
/usr/lib64/python3.8/config-3.8-x86_64-linux-gnu

Comment 7 Miro Hrončok 2019-05-20 10:48:23 UTC
This is  deliberate change by Python upstream.

https://docs.python.org/dev/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build

CCing Victor as well.

Comment 8 Miro Hrončok 2019-05-20 11:02:21 UTC
Do you actually embed Python or so you only build a Python extension module?

If you only build an extension module, you don't need to link against -lpython3.8 and you should go with the same fix as wxpython4 did (as linked in the bug description).


https://src.fedoraproject.org/rpms/python-wxpython4/c/18d4c1babd89a31e555cf2cbb0345b96ba0b87ac?branch=master

Comment 9 Lukas Slebodnik 2019-05-20 11:26:39 UTC
(In reply to Miro Hrončok from comment #8)
> Do you actually embed Python or so you only build a Python extension module?
> 
> If you only build an extension module, you don't need to link against
> -lpython3.8 and you should go with the same fix as wxpython4 did (as linked
> in the bug description).
> 
> 
> https://src.fedoraproject.org/rpms/python-wxpython4/c/
> 18d4c1babd89a31e555cf2cbb0345b96ba0b87ac?branch=master

Patch is slightly different but it seems to help

diff --git a/third_party/waf/waflib/Tools/python.py b/third_party/waf/waflib/Tools/python.py
index 52a05c668e3..e9f719c236c 100644
--- a/third_party/waf/waflib/Tools/python.py
+++ b/third_party/waf/waflib/Tools/python.py
@@ -282,7 +282,7 @@ def python_cross_compile(self, features='pyembed pyext'):
        return True
 
 @conf
-def check_python_headers(conf, features='pyembed pyext'):
+def check_python_headers(conf, features='pyext'):
        """
        Check for headers and libraries necessary to extend or embed python by using the module *distutils*.
        On success the environment variables xxx_PYEXT and xxx_PYEMBED are added:


I would appreciate if you could help with explanation for removing pyembed from default feature.
I am not sure what is it was used for and what would a difference for older versions of python(2,3)

Comment 10 Miro Hrončok 2019-05-20 11:35:56 UTC
There are several things to consider here:

* waf allows to "find and check Python" for extension modules and embedding, utilizing both checks by default
* this package only calls check_python_headers() without arguments, hence it checks Python for both extension modules and embedding
* the way waf checks for embedded flags is currently most likely broken (I don't know if that way is supposed to work)

So there are probably 2 problems that only manifests when both happen:

 - this package checks for embedded flags when it doesn't do any embedded
 - python3.8-config cannot properly check for embedded flags (speculation)

Comment 11 Lukas Slebodnik 2019-05-20 11:43:24 UTC
(In reply to Miro Hrončok from comment #10)
> There are several things to consider here:
> 
> * waf allows to "find and check Python" for extension modules and embedding,
> utilizing both checks by default
> * this package only calls check_python_headers() without arguments, hence it
> checks Python for both extension modules and embedding
> * the way waf checks for embedded flags is currently most likely broken (I
> don't know if that way is supposed to work)
> 
> So there are probably 2 problems that only manifests when both happen:
> 
>  - this package checks for embedded flags when it doesn't do any embedded
>  - python3.8-config cannot properly check for embedded flags (speculation)

Will try to write summary to commit message. Hopefully it will be acceptable to upstream.

BTW the same problem will be in also in libtalloc, libtevent, libldb and samba

Comment 12 Miro Hrončok 2019-05-20 11:56:45 UTC
> BTW the same problem will be in also in libtalloc, libtevent, libldb and samba

I've suspected that.

Comment 13 Miro Hrončok 2019-05-20 12:00:25 UTC
As a matter of style, I think changing the conf.check_python_headers() call in buildtools/wafsamba/samba_python.py makes more sense than changing the default value in waf's check_python_headers() definition.

Comment 14 Miro Hrončok 2019-05-20 12:03:21 UTC
I've also found https://bugs.python.org/issue36721

Comment 15 Miro Hrončok 2019-05-21 22:47:38 UTC
https://src.fedoraproject.org/rpms/libtdb/pull-request/3 gets the job done.

Comment 16 Lukas Slebodnik 2019-05-21 23:35:44 UTC
(In reply to Miro Hrončok from comment #13)
> As a matter of style, I think changing the conf.check_python_headers() call
> in buildtools/wafsamba/samba_python.py makes more sense than changing the
> default value in waf's check_python_headers() definition.

sorry for late reply but it is just a workaround in libtdb.
But the same change does not work in libtalloc and libldb they need both of them.
(and very likely also for samba itself)

e.g.
pyext is required for /usr/lib64/python3.7/site-packages/talloc.cpython-37m-s390x-linux-gnu.so
and pyembed for /usr/lib64/libpytalloc-util.cpython-37m-s390x-linux-gnu.so.2.1.16

The library libpytalloc-util. ... .2.1.6 is linked with files in other packages

sh# dnf repoquery --whatrequires 'libpytalloc-util.cpython-37m-x86-64-linux-gnu.so.2()(64bit)'
Last metadata expiration check: 0:16:29 ago on Wed 22 May 2019 01:04:53 AM CEST.
python3-samba-2:4.10.3-0.fc31.x86_64
python3-samba-dc-2:4.10.3-0.fc31.x86_64
python3-talloc-devel-0:2.1.16-1.fc31.x86_64

And thus build failed.

../../pytalloc_util.c:20:10: fatal error: Python.h: No such file or directory
   20 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.

../../pytalloc_util.c:20:10: fatal error: Python.h: No such file or directory
   20 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
Waf: Leaving directory `/builddir/build/BUILD/talloc-2.1.16/bin/default'
Build failed
 -> task in 'pytalloc-util.cpython-38-x86_64-linux-gnu.objlist' failed with exit status 1: 
        {task 140431097927256: c pytalloc_util.c -> pytalloc_util.c.13.o}
['/usr/bin/gcc', '-O2', '-g', '-pipe', '-Wall', '-Werror=format-security', '-Wp,-D_FORTIFY_SOURCE=2', '-Wp,-D_GLIBCXX_ASSERTIONS', '-fexceptions', '-fstack-protector-strong', '-grecord-gcc-switches', '-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1', '-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1', '-m64', '-mtune=generic', '-fasynchronous-unwind-tables', '-fstack-clash-protection', '-fcf-protection', '-MMD', '-D_GNU_SOURCE=1', '-D_XOPEN_SOURCE_EXTENDED=1', '-fPIC', '-D__STDC_WANT_LIB_EXT1__=1', '-fstack-protector-strong', '-fstack-clash-protection', '-fvisibility=hidden', '-I.', '-I../..', '-I.', '-I../..', '-Ilib/replace', '-I../../lib/replace', '-DHAVE_PYEXT=1', '-DHAVE_PYTHON_H=1', '../../pytalloc_util.c', '-c', '-o/builddir/build/BUILD/talloc-2.1.16/bin/default/pytalloc_util.c.13.o']
 -> task in 'pytalloc-util.objlist' failed with exit status 1: 
        {task 140431097928192: c pytalloc_util.c -> pytalloc_util.c.23.o}
['/usr/bin/gcc', '-O2', '-g', '-pipe', '-Wall', '-Werror=format-security', '-Wp,-D_FORTIFY_SOURCE=2', '-Wp,-D_GLIBCXX_ASSERTIONS', '-fexceptions', '-fstack-protector-strong', '-grecord-gcc-switches', '-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1', '-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1', '-m64', '-mtune=generic', '-fasynchronous-unwind-tables', '-fstack-clash-protection', '-fcf-protection', '-MMD', '-D_GNU_SOURCE=1', '-D_XOPEN_SOURCE_EXTENDED=1', '-fPIC', '-D__STDC_WANT_LIB_EXT1__=1', '-fstack-protector-strong', '-fstack-clash-protection', '-fvisibility=hidden', '-I.', '-I../..', '-I.', '-I../..', '-Ilib/replace', '-I../../lib/replace', '-DPYTHONDIR="/usr/lib/python2.7/site-packages"', '-DPYTHONARCHDIR="/usr/lib64/python2.7/site-packages"', '-DHAVE_PYEXT=1', '-DHAVE_PYTHON_H=1', '../../pytalloc_util.c', '-c', '-o/builddir/build/BUILD/talloc-2.1.16/bin/default/pytalloc_util.c.23.o']
make: *** [Makefile:8: all] Error 1
error: Bad exit status from /var/tmp/rpm-tmp.NyjjQv (%build)

Comment 17 Lukas Slebodnik 2019-05-21 23:49:08 UTC
I tried to do some POC changes to use pkg-config instead of python3-config for pyembed
e.g.

diff --git a/third_party/waf/waflib/Tools/python.py b/third_party/waf/waflib/Tools/python.py
index 52a05c668e3..2e80cd4901f 100644
--- a/third_party/waf/waflib/Tools/python.py
+++ b/third_party/waf/waflib/Tools/python.py
@@ -337,8 +337,9 @@ def check_python_headers(conf, features='pyembed pyext'):
                xx = env.CXX_NAME and 'cxx' or 'c'
 
                if 'pyembed' in features:
-                       for flags in all_flags:
-                               conf.check_cfg(msg='Asking python-config for pyembed %r flags' % ' '.join(flags), path=env.PYTHON_CONFIG, package='', uselib_store='PYEMBED', args=flags)
+                       for flags in [['--cflags', '--libs']]:
+                               conf.check_cfg(msg='Asking pkg-config for pyembed %r flags' % ' '.join(flags), package="python-%s" % env.PYTHON_VERSION, uselib_store='PYEMBED', args=flags)
+
 
                        try:
                                conf.test_pyembed(xx)


but -lpython3 was used even for pyext and not just for pyembed

[build@host mock]$  grep lpython3 build.log 
01:38:42 runner ['/usr/bin/gcc', '-Wl,--version-script=/builddir/build/BUILD/talloc-2.1.16/bin/default/pytalloc-util.cpython-38-x86_64-linux-gnu.vscript', '-shared', '-Wl,-h,libpytalloc-util.cpython-38-x86-64-linux-gnu.so.2', 'pytalloc_util.c.13.o', '-o/builddir/build/BUILD/talloc-2.1.16/bin/default/libpytalloc-util.cpython-38-x86-64-linux-gnu.so', '-Wl,-Bstatic', '-Wl,-Bdynamic', '-L/builddir/build/BUILD/talloc-2.1.16/bin/default', '-L/usr/local/lib', '-ltalloc', '-ldl', '-lcrypt', '-lpython3.8', '-Wl,-z,relro', '-Wl,--as-needed', '-Wl,-z,now', '-specs=/usr/lib/rpm/redhat/redhat-hardened-ld', '-Wl,-no-undefined']
01:38:42 runner ['/usr/bin/gcc', '-shared', '-fPIC', '-fPIC', 'pytalloc.c.19.o', '-o/builddir/build/BUILD/talloc-2.1.16/bin/default/pytalloc.cpython-38-x86_64-linux-gnu.so', '-Wl,-Bstatic', '-Wl,-Bdynamic', '-L/builddir/build/BUILD/talloc-2.1.16/bin/default', '-L/usr/local/lib', '-L/usr/lib64', '-lpytalloc-util.cpython-38-x86-64-linux-gnu', '-ltalloc', '-lpython3.8', '-lcrypt', '-ldl', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-Wl,-z,relro', '-Wl,--as-needed', '-Wl,-z,now', '-specs=/usr/lib/rpm/redhat/redhat-hardened-ld']
01:38:42 runner ['/usr/bin/gcc', '-shared', '-fPIC', '-fPIC', 'test_pytalloc.c.21.o', '-o/builddir/build/BUILD/talloc-2.1.16/bin/default/test-pytalloc.cpython-38-x86_64-linux-gnu.so', '-Wl,-Bstatic', '-Wl,-Bdynamic', '-L/builddir/build/BUILD/talloc-2.1.16/bin/default', '-L/usr/local/lib', '-L/usr/lib64', '-lpytalloc-util.cpython-38-x86-64-linux-gnu', '-ltalloc', '-lpython3.8', '-lcrypt', '-ldl', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-Wl,-z,relro', '-Wl,--as-needed', '-Wl,-z,now', '-specs=/usr/lib/rpm/redhat/redhat-hardened-ld']

and -lpython3 should not be used for '-o/builddir/build/BUILD/talloc-2.1.16/bin/default/pytalloc.cpython-38-x86_64-linux-gnu.so'

Miro,
I would appreciate a help here cause I do not have more time to fight with waf (third_party/waf in samba upstream) and usage of waf in samba(buildtools/wafsamba in samba upstream)

Comment 18 Miro Hrončok 2019-05-21 23:51:44 UTC
Victor, how do we deal with this breakage? Is there at least some viable workaround?

Comment 19 Andreas Schneider 2019-05-22 08:47:26 UTC
I've pinged the waf maintainer. We need a proper fix upstream.

Comment 20 Andreas Schneider 2019-05-22 11:35:24 UTC
Miro: Fix python3-config and everything will be fine.

Comment 21 Miro Hrončok 2019-05-22 11:51:52 UTC
Andreas: How exactly do I fix it?

This is a complex problem: waf uses it both for extension modules and embedded. If we revert the change, we link Python extensions against libpython and that should no longer be done.

Comment 22 Lukas Slebodnik 2019-05-22 12:44:07 UTC
(In reply to Lukas Slebodnik from comment #17)
> I tried to do some POC changes to use pkg-config instead of python3-config
> for pyembed
> e.g.
> 
> diff --git a/third_party/waf/waflib/Tools/python.py
> b/third_party/waf/waflib/Tools/python.py
> index 52a05c668e3..2e80cd4901f 100644
> --- a/third_party/waf/waflib/Tools/python.py
> +++ b/third_party/waf/waflib/Tools/python.py
> @@ -337,8 +337,9 @@ def check_python_headers(conf, features='pyembed pyext'):
>                 xx = env.CXX_NAME and 'cxx' or 'c'
>  
>                 if 'pyembed' in features:
> -                       for flags in all_flags:
> -                               conf.check_cfg(msg='Asking python-config for
> pyembed %r flags' % ' '.join(flags), path=env.PYTHON_CONFIG, package='',
> uselib_store='PYEMBED', args=flags)
> +                       for flags in [['--cflags', '--libs']]:
> +                               conf.check_cfg(msg='Asking pkg-config for
> pyembed %r flags' % ' '.join(flags), package="python-%s" %
> env.PYTHON_VERSION, uselib_store='PYEMBED', args=flags)
> +
>  
>                         try:
>                                 conf.test_pyembed(xx)
> 

Previous patch (using pkg-config for pyembed) works well for libtdb.


But it does not work for libtalloc(pytalloc.cpython-38-x86_64-linux-gnu.so)
because pytalloc.so has two dependencies libtalloc and libpytalloc-util

And it seems that all libs necessary for libpytalloc-util
were used also for linking pytalloc.so
(autotools/libtool call such feature link_all_deplibs)

I am not syre whether it is possible to avoid such behaviour in waf.


> 
> but -lpython3 was used even for pyext and not just for pyembed
> 
> [build@host mock]$  grep lpython3 build.log 
> 01:38:42 runner ['/usr/bin/gcc',
> '-Wl,--version-script=/builddir/build/BUILD/talloc-2.1.16/bin/default/
> pytalloc-util.cpython-38-x86_64-linux-gnu.vscript', '-shared',
> '-Wl,-h,libpytalloc-util.cpython-38-x86-64-linux-gnu.so.2',
> 'pytalloc_util.c.13.o',
> '-o/builddir/build/BUILD/talloc-2.1.16/bin/default/libpytalloc-util.cpython-
> 38-x86-64-linux-gnu.so', '-Wl,-Bstatic', '-Wl,-Bdynamic',
> '-L/builddir/build/BUILD/talloc-2.1.16/bin/default', '-L/usr/local/lib',
> '-ltalloc', '-ldl', '-lcrypt', '-lpython3.8', '-Wl,-z,relro',
> '-Wl,--as-needed', '-Wl,-z,now',
> '-specs=/usr/lib/rpm/redhat/redhat-hardened-ld', '-Wl,-no-undefined']
> 01:38:42 runner ['/usr/bin/gcc', '-shared', '-fPIC', '-fPIC',
> 'pytalloc.c.19.o',
> '-o/builddir/build/BUILD/talloc-2.1.16/bin/default/pytalloc.cpython-38-
> x86_64-linux-gnu.so', '-Wl,-Bstatic', '-Wl,-Bdynamic',
> '-L/builddir/build/BUILD/talloc-2.1.16/bin/default', '-L/usr/local/lib',
> '-L/usr/lib64', '-lpytalloc-util.cpython-38-x86-64-linux-gnu', '-ltalloc',
> '-lpython3.8', '-lcrypt', '-ldl', '-lcrypt', '-lpthread', '-ldl', '-lutil',
   ^^^^^^^^^^^^

What will happen if python extension is linked with this library ?

> '-lm', '-lm', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm',
> '-Wl,-z,relro', '-Wl,--as-needed', '-Wl,-z,now',
> '-specs=/usr/lib/rpm/redhat/redhat-hardened-ld']

Comment 23 Miro Hrončok 2019-05-22 12:58:49 UTC
> What will happen if python extension is linked with -lpython3.8?

The following 3.8 feature stops working:

https://docs.python.org/dev/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build

Of course we could probably deliberately make a choice to temporarily ditch the future for libtalloc, libtevent, libldb and samba to make it build, but we should not revert that feature globally (or at least we should not take that decision lightly).

Do we know how to make this work for the samba packages at least?

Comment 24 Lukas Slebodnik 2019-05-22 13:34:06 UTC
(In reply to Miro Hrončok from comment #23)
> > What will happen if python extension is linked with -lpython3.8?
> 
> The following 3.8 feature stops working:
> 
> https://docs.python.org/dev/whatsnew/3.8.html#debug-build-uses-the-same-abi-
> as-release-build
> 

I read that section few times and I am not shure what will happen if extension is linked with
libpython3.8

But build passed for me including sanity python tests and it works even
after installation rpms.


<mock-chroot> sh-5.0# python3
Python 3.8.0a4 (default, May 17 2019, 12:33:06) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import talloc
>>> talloc.total_blocks()
0
>>> talloc.enable_null_tracking()
>>> talloc.total_blocks()
1
>>> 
<mock-chroot> sh-5.0# objdump -p /usr/lib64/python3.8/site-packages/talloc.cpython-38-x86_64-linux-gnu.so | grep NEEDED
  NEEDED               libpytalloc-util.cpython-38-x86-64-linux-gnu.so.2
  NEEDED               libtalloc.so.2
  NEEDED               libpython3.8.so.1.0
  NEEDED               libcrypt.so.2
  NEEDED               libdl.so.2
  NEEDED               libpthread.so.0
  NEEDED               libutil.so.1
  NEEDED               libm.so.6
  NEEDED               libc.so.6



> Of course we could probably deliberately make a choice to temporarily ditch
> the future for libtalloc, libtevent, libldb and samba to make it build, but
> we should not revert that feature globally (or at least we should not take
> that decision lightly).
> 
> Do we know how to make this work for the samba packages at least?

no idea

Comment 25 Victor Stinner 2019-05-22 13:53:37 UTC
It seems like one root issue is that waf "pyembed" fails to detect Python 3.8. Internally, waf creates a C program which runs Py_Initialize(), but the linker fails to find Py_Initialize() because "python3.8-config --libs" output no longer contains -lpython3.8.

I reported the issue to waf:
https://gitlab.com/ita1024/waf/issues/2239

Fixing waf & other programs requires to add a new --embd option to python3.8-config and pkg-config python3.8 to include -lpython3.8. I'm working on implementing https://bugs.python.org/issue36721

Comment 26 Andreas Schneider 2019-05-22 15:25:21 UTC
First we need to fix waf, then import the new waf version into Samba. Once we have that, we can fix wafsamba.

Comment 27 Miro Hrončok 2019-05-24 09:46:16 UTC
*** Bug 1712602 has been marked as a duplicate of this bug. ***

Comment 28 Miro Hrončok 2019-05-24 09:55:01 UTC
Created attachment 1572856 [details]
Patch for all libtdb, libtalloc, libtevent, libldb and samba

Assigned a patch that applies to all the affected packages.

Running Copr builds of all of them, got as far as libldb for now.

Submitted upstream: https://gitlab.com/ita1024/waf/merge_requests/2236

Comment 29 Miro Hrončok 2019-05-24 09:57:25 UTC
> Assigned a patch

Attached

Comment 30 Lukas Slebodnik 2019-05-24 14:30:45 UTC
(In reply to Miro Hrončok from comment #29)
> > Assigned a patch
> 
> Attached

Thank you very much. It works in similar way as POC version with using pkg-config (comment 17)

But python extension is still linked with -lpython3.8

[build@host mock]$ grep lpython3 build.log 
16:20:06 runner ['/usr/bin/gcc', '-Wl,--version-script=/builddir/build/BUILD/talloc-2.1.16/bin/default/pytalloc-util.cpython-38-x86_64-linux-gnu.vscript', '-shared', '-fPIC', '-fPIC', '-Wl,-h,libpytalloc-util.cpython-38-x86-64-linux-gnu.so.2', 'pytalloc_util.c.13.o', '-o/builddir/build/BUILD/talloc-2.1.16/bin/default/libpytalloc-util.cpython-38-x86-64-linux-gnu.so', '-Wl,-Bstatic', '-Wl,-Bdynamic', '-L/builddir/build/BUILD/talloc-2.1.16/bin/default', '-L/usr/local/lib', '-ltalloc', '-ldl', '-lcrypt', '-lpython3.8', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-lpython3.8', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-Wl,-z,relro', '-Wl,--as-needed', '-Wl,-z,now', '-specs=/usr/lib/rpm/redhat/redhat-hardened-ld', '-Wl,-no-undefined']
16:20:06 runner ['/usr/bin/gcc', '-shared', '-fPIC', '-fPIC', '-fPIC', '-fPIC', 'pytalloc.c.19.o', '-o/builddir/build/BUILD/talloc-2.1.16/bin/default/pytalloc.cpython-38-x86_64-linux-gnu.so', '-Wl,-Bstatic', '-Wl,-Bdynamic', '-L/builddir/build/BUILD/talloc-2.1.16/bin/default', '-L/usr/local/lib', '-L/usr/lib64', '-L/usr/lib64', '-lpytalloc-util.cpython-38-x86-64-linux-gnu', '-ltalloc', '-lpython3.8', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-lpython3.8', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-lcrypt', '-ldl', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-Wl,-z,relro', '-Wl,--as-needed', '-Wl,-z,now', '-specs=/usr/lib/rpm/redhat/redhat-hardened-ld']
16:20:06 runner ['/usr/bin/gcc', '-shared', '-fPIC', '-fPIC', '-fPIC', '-fPIC', 'test_pytalloc.c.21.o', '-o/builddir/build/BUILD/talloc-2.1.16/bin/default/test-pytalloc.cpython-38-x86_64-linux-gnu.so', '-Wl,-Bstatic', '-Wl,-Bdynamic', '-L/builddir/build/BUILD/talloc-2.1.16/bin/default', '-L/usr/local/lib', '-L/usr/lib64', '-L/usr/lib64', '-lpytalloc-util.cpython-38-x86-64-linux-gnu', '-ltalloc', '-lpython3.8', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-lpython3.8', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-lcrypt', '-ldl', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-Wl,-z,relro', '-Wl,--as-needed', '-Wl,-z,now', '-specs=/usr/lib/rpm/redhat/redhat-hardened-ld']


<mock-chroot> sh-5.0# objdump -p /builddir/build/BUILD/talloc-2.1.16/bin/default/pytalloc.cpython-38-x86_64-linux-gnu.so | grep NEEDED
  NEEDED               libpytalloc-util.cpython-38-x86-64-linux-gnu.so.2
  NEEDED               libtalloc.so.2
  NEEDED               libpython3.8.so.1.0
  NEEDED               libcrypt.so.2
  NEEDED               libpthread.so.0
  NEEDED               libdl.so.2
  NEEDED               libutil.so.1
  NEEDED               libm.so.6
  NEEDED               libc.so.6


It seems to be caused by linking extensions with all dependencies of libpytalloc-util.so
(as I already mentioned in comment 22)

I will be fine with applying patch in fedora if it does not cause any problem for python3.8
(BTW python test passed in %check phase)

Comment 31 Miro Hrončok 2019-05-24 15:01:44 UTC
IMHO linking the extensions against libpython is not a very severe issue.

Comment 32 Lukas Slebodnik 2019-05-27 09:39:54 UTC
(In reply to Miro Hrončok from comment #31)
> IMHO linking the extensions against libpython is not a very severe issue.

Great

Victor, can you confirm the same ?
I'll rather double check.

Comment 33 Victor Stinner 2019-06-03 15:42:49 UTC
Linking an extension to libpython is perfectly safe. Python always did that for all C extensions on all platforms. Only Python 3.8 started to no longer link C extensions to libpython on some platforms: Unix except Android and Cygwin.
https://docs.python.org/dev/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build

libtalloc and libtdb can be linked to libpython.

The main drawback is that you loose the ability to use a debug build of Python with libtalloc and libtdb compiled in release mode. It's a minor issue ;-)

Comment 34 Lukas Slebodnik 2019-06-03 20:24:31 UTC
(In reply to Victor Stinner from comment #33)
> Linking an extension to libpython is perfectly safe. Python always did that
> for all C extensions on all platforms. Only Python 3.8 started to no longer
> link C extensions to libpython on some platforms: Unix except Android and
> Cygwin.
> https://docs.python.org/dev/whatsnew/3.8.html#debug-build-uses-the-same-abi-
> as-release-build
> 
> libtalloc and libtdb can be linked to libpython.
> 
> The main drawback is that you loose the ability to use a debug build of
> Python with libtalloc and libtdb compiled in release mode. It's a minor
> issue ;-)

Just libtalloc and libldb will be linked with lpython3.8 (due to nested lining done in waf
libtevent and libtdb are fine.

Comment 35 Miro Hrončok 2019-06-03 21:33:33 UTC
I hope the patch is now considered OK. let me know if I shall send a PR, commit directly or if you prefer to this yourself.

Comment 36 Lukas Slebodnik 2019-06-04 07:30:43 UTC
(In reply to Miro Hrončok from comment #35)
> I hope the patch is now considered OK. let me know if I shall send a PR,
> commit directly or if you prefer to this yourself.

Patches+builds are already in rawhide for lib{talloc, tevent, tdb, ldb}

Andreas,
could you add patch to samba dist-git ?

https://bugzilla.redhat.com/attachment.cgi?id=1572856

Comment 37 Victor Stinner 2019-06-05 09:45:58 UTC
> Andreas, could you add patch to samba dist-git ?
> https://bugzilla.redhat.com/attachment.cgi?id=1572856

FYI Miro made this change upstream in waf and waf 2.0.17 has been released with the fix.

Comment 38 Andreas Schneider 2019-06-05 12:01:13 UTC
waf update MR for Samba: https://gitlab.com/samba-team/samba/merge_requests/519

Once that is in I will backport to the 4.10 branch. We have to do new releases of Samba and the libs then.

Comment 39 Lukas Slebodnik 2019-06-05 17:38:28 UTC
(In reply to Andreas Schneider from comment #38)
> waf update MR for Samba:
> https://gitlab.com/samba-team/samba/merge_requests/519
> 
> Once that is in I will backport to the 4.10 branch. We have to do new
> releases of Samba and the libs then.

Have you consider patching it in dist-git meanwhile?
IMHO we needn't wait for next regular 4.10.x release

Comment 40 Miro Hrončok 2019-06-06 23:41:04 UTC
See also bz1718113.

Comment 41 Miro Hrončok 2019-06-10 16:44:22 UTC
This issue is blocking the Python 3.8 rebuilds. If this package won't build with 3.8, it won't be installable, along with all its dependent packages, after the side tag is merged.
Furthermore, as it fails to install, its dependent packages will fail to install and/or build as well. The fix should be pushed on the master branch and no release bump is required.

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.8: https://copr.fedorainfracloud.org/coprs/g/python/python3.8/

This issue needs to be resolved by next week, if other rebuilds of Python 3.8 beta 1 go well. If this is unrealistic for you, let us know how much time you need.

If you don't have free cycles to dedicate fixing your package, notify us and we'll try to provide some pointers. Let us know if we can push a fix directly without a pull request, in the case we happen to have one before you do.

We recommend always consulting with upstream, chances are this is already fixed there.

Comment 42 Miro Hrončok 2019-06-17 09:45:13 UTC
This issue is blocking the Python 3.8 rebuilds. This package is critical for Fedora, unless this is resolved, we cannot proceed with the side tag merge.

Comment 43 Andreas Schneider 2019-06-17 11:43:57 UTC
https://bugzilla.samba.org/show_bug.cgi?id=13960

Comment 44 Miro Hrončok 2019-06-17 12:02:52 UTC
(In reply to Lukas Slebodnik from comment #39)
> Have you consider patching it in dist-git meanwhile?
> IMHO we needn't wait for next regular 4.10.x release

Comment 45 Andreas Schneider 2019-06-17 14:22:21 UTC
With the above patch you can patch samba and do it. However there is also bug #1718113 which needs to be fixed. Waiting for upstream to accept it.

Comment 46 Lukas Slebodnik 2019-06-17 16:36:42 UTC
(In reply to Andreas Schneider from comment #45)
> With the above patch you can patch samba and do it. However there is also
> bug #1718113 which needs to be fixed. Waiting for upstream to accept it.

Issue with "time.clock()" in BZ#1718113 is not critical. Because it is used just in debug messages and does not have any significant impact on buildsystem. So even removing lines with problematic debug messages would fix the problem.

Do you really want to wait for upstream?

Comment 47 Lukas Slebodnik 2019-06-19 10:05:30 UTC
(In reply to Andreas Schneider from comment #45)
> With the above patch you can patch samba and do it. However there is also
> bug #1718113 which needs to be fixed. Waiting for upstream to accept it.

Patch for BZ#1718113 is already in upstream 
https://gitlab.com/samba-team/samba/merge_requests/546

Is there any blocker to backport patches to rawhide?

Comment 48 Miro Hrončok 2019-06-19 10:18:52 UTC
(In reply to Lukas Slebodnik from comment #47)
> Is there any blocker to backport patches to rawhide?

There was, however Python 3.8 will most likely be deferred to Fedora 32, so there is not right now.

That said, having the patch applied in master will make our work significantly easier.

Comment 49 Miro Hrončok 2019-06-30 00:14:09 UTC
Can we please get this fixed?

Comment 50 Andreas Schneider 2019-07-01 18:05:04 UTC
Pushed as 1ac5f2f4b603b67eefbde338c9d96d633ad4b96d


Note You need to log in before you can comment on or make changes to this bug.