Created attachment 1570816 [details] Full log from Copr libtdb-1.3.18-3.fc31 fails to build with 3.8.0a4 in https://copr.fedorainfracloud.org/coprs/g/python/python3.8/ Testing pyembed configuration : Could not build a python embedded interpreter Testing pyembed configuration : Could not build a python embedded interpreter ... Traceback (most recent call last): File "python/tests/simple.py", line 14, in <module> import tdb ImportError: No module named tdb Traceback (most recent call last): File "python/tests/simple.py", line 14, in <module> import tdb ModuleNotFoundError: No module named 'tdb' ... Running Python test with /usr/bin/python2: python/tests/simple.py Python test failed: PYTHONPATH=bin/python /usr/bin/python2 python/tests/simple.py Running Python test with /usr/bin/python3: python/tests/simple.py Python test failed: PYTHONPATH=bin/python /usr/bin/python3 python/tests/simple.py python testsuite returned 1 make: *** [Makefile:17: test] Error 1 error: Bad exit status from /var/tmp/rpm-tmp.ToUNLv (%check) Full log attached. See bz1710767 for a similar problem (fixed) in python-wxpython4.
There are dependencies between libraries provided by samba libtdb -> libldb libtalloc -> libtevent -> libldb And I cannot see good build for libtdb https://copr.fedorainfracloud.org/coprs/g/python/python3.8/build/911193/
You cannot see a good build of libtdb, because this is a bug for libtdb failing to build.
(In reply to Miro Hrončok from comment #2) > You cannot see a good build of libtdb, because this is a bug for libtdb > failing to build. Sorry, I misread libldb/libtdb Will try do debug in mock later today.
> Sorry, I misread libldb/libtdb Happens to me all the time.
I would say there is some isseu with python3-config It does not return -lpython3.8 among linker flags <mock-chroot> sh-5.0# /usr/bin/python3-config --cflags --libs --ldflags -I/usr/include/python3.8 -I/usr/include/python3.8 -Wno-unused-result -Wsign-compare -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -lcrypt -lpthread -ldl -lutil -lm -lm -L/usr/lib64 -lcrypt -lpthread -ldl -lutil -lm -lm python2-config works well <mock-chroot> sh-5.0# /usr/bin/python2-config --cflags --libs --ldflags -I/usr/include/python2.7 -I/usr/include/python2.7 -fno-strict-aliasing -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -lpython2.7 -lpthread -ldl -lutil -lm -lpython2.7 -lpthread -ldl -lutil -lm -Xlinker -export-dynamic
<mock-chroot> sh-5.0# /usr/bin/python3-config -h Usage: /usr/bin/python3.8-x86_64-config --prefix|--exec-prefix|--includes|--libs|--cflags|--ldflags|--extension-suffix|--help|--abiflags|--configdir <mock-chroot> sh-5.0# /usr/bin/python2-config Usage: /usr/bin/python2-config [--prefix|--exec-prefix|--includes|--libs|--cflags|--ldflags|--extension-suffix|--help] <mock-chroot> sh-5.0# /usr/bin/python2-config --extension-suffix .so <mock-chroot> sh-5.0# /usr/bin/python3-config --extension-suffix .cpython-38-x86_64-linux-gnu.so <mock-chroot> sh-5.0# /usr/bin/python3-config --prefix /usr <mock-chroot> sh-5.0# /usr/bin/python3-config --exec-prefix /usr <mock-chroot> sh-5.0# /usr/bin/python3-config --includes -I/usr/include/python3.8 -I/usr/include/python3.8 <mock-chroot> sh-5.0# /usr/bin/python3-config --libs -lcrypt -lpthread -ldl -lutil -lm -lm <mock-chroot> sh-5.0# /usr/bin/python3-config --cflags -I/usr/include/python3.8 -I/usr/include/python3.8 -Wno-unused-result -Wsign-compare -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv <mock-chroot> sh-5.0# /usr/bin/python3-config --ldflags -L/usr/lib64 -lcrypt -lpthread -ldl -lutil -lm -lm <mock-chroot> sh-5.0# /usr/bin/python3-config --extension-suffix .cpython-38-x86_64-linux-gnu.so <mock-chroot> sh-5.0# /usr/bin/python3-config --abiflags <mock-chroot> sh-5.0# /usr/bin/python3-config --configdir /usr/lib64/python3.8/config-3.8-x86_64-linux-gnu
This is deliberate change by Python upstream. https://docs.python.org/dev/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build CCing Victor as well.
Do you actually embed Python or so you only build a Python extension module? If you only build an extension module, you don't need to link against -lpython3.8 and you should go with the same fix as wxpython4 did (as linked in the bug description). https://src.fedoraproject.org/rpms/python-wxpython4/c/18d4c1babd89a31e555cf2cbb0345b96ba0b87ac?branch=master
(In reply to Miro Hrončok from comment #8) > Do you actually embed Python or so you only build a Python extension module? > > If you only build an extension module, you don't need to link against > -lpython3.8 and you should go with the same fix as wxpython4 did (as linked > in the bug description). > > > https://src.fedoraproject.org/rpms/python-wxpython4/c/ > 18d4c1babd89a31e555cf2cbb0345b96ba0b87ac?branch=master Patch is slightly different but it seems to help diff --git a/third_party/waf/waflib/Tools/python.py b/third_party/waf/waflib/Tools/python.py index 52a05c668e3..e9f719c236c 100644 --- a/third_party/waf/waflib/Tools/python.py +++ b/third_party/waf/waflib/Tools/python.py @@ -282,7 +282,7 @@ def python_cross_compile(self, features='pyembed pyext'): return True @conf -def check_python_headers(conf, features='pyembed pyext'): +def check_python_headers(conf, features='pyext'): """ Check for headers and libraries necessary to extend or embed python by using the module *distutils*. On success the environment variables xxx_PYEXT and xxx_PYEMBED are added: I would appreciate if you could help with explanation for removing pyembed from default feature. I am not sure what is it was used for and what would a difference for older versions of python(2,3)
There are several things to consider here: * waf allows to "find and check Python" for extension modules and embedding, utilizing both checks by default * this package only calls check_python_headers() without arguments, hence it checks Python for both extension modules and embedding * the way waf checks for embedded flags is currently most likely broken (I don't know if that way is supposed to work) So there are probably 2 problems that only manifests when both happen: - this package checks for embedded flags when it doesn't do any embedded - python3.8-config cannot properly check for embedded flags (speculation)
(In reply to Miro Hrončok from comment #10) > There are several things to consider here: > > * waf allows to "find and check Python" for extension modules and embedding, > utilizing both checks by default > * this package only calls check_python_headers() without arguments, hence it > checks Python for both extension modules and embedding > * the way waf checks for embedded flags is currently most likely broken (I > don't know if that way is supposed to work) > > So there are probably 2 problems that only manifests when both happen: > > - this package checks for embedded flags when it doesn't do any embedded > - python3.8-config cannot properly check for embedded flags (speculation) Will try to write summary to commit message. Hopefully it will be acceptable to upstream. BTW the same problem will be in also in libtalloc, libtevent, libldb and samba
> BTW the same problem will be in also in libtalloc, libtevent, libldb and samba I've suspected that.
As a matter of style, I think changing the conf.check_python_headers() call in buildtools/wafsamba/samba_python.py makes more sense than changing the default value in waf's check_python_headers() definition.
I've also found https://bugs.python.org/issue36721
https://src.fedoraproject.org/rpms/libtdb/pull-request/3 gets the job done.
(In reply to Miro Hrončok from comment #13) > As a matter of style, I think changing the conf.check_python_headers() call > in buildtools/wafsamba/samba_python.py makes more sense than changing the > default value in waf's check_python_headers() definition. sorry for late reply but it is just a workaround in libtdb. But the same change does not work in libtalloc and libldb they need both of them. (and very likely also for samba itself) e.g. pyext is required for /usr/lib64/python3.7/site-packages/talloc.cpython-37m-s390x-linux-gnu.so and pyembed for /usr/lib64/libpytalloc-util.cpython-37m-s390x-linux-gnu.so.2.1.16 The library libpytalloc-util. ... .2.1.6 is linked with files in other packages sh# dnf repoquery --whatrequires 'libpytalloc-util.cpython-37m-x86-64-linux-gnu.so.2()(64bit)' Last metadata expiration check: 0:16:29 ago on Wed 22 May 2019 01:04:53 AM CEST. python3-samba-2:4.10.3-0.fc31.x86_64 python3-samba-dc-2:4.10.3-0.fc31.x86_64 python3-talloc-devel-0:2.1.16-1.fc31.x86_64 And thus build failed. ../../pytalloc_util.c:20:10: fatal error: Python.h: No such file or directory 20 | #include <Python.h> | ^~~~~~~~~~ compilation terminated. ../../pytalloc_util.c:20:10: fatal error: Python.h: No such file or directory 20 | #include <Python.h> | ^~~~~~~~~~ compilation terminated. Waf: Leaving directory `/builddir/build/BUILD/talloc-2.1.16/bin/default' Build failed -> task in 'pytalloc-util.cpython-38-x86_64-linux-gnu.objlist' failed with exit status 1: {task 140431097927256: c pytalloc_util.c -> pytalloc_util.c.13.o} ['/usr/bin/gcc', '-O2', '-g', '-pipe', '-Wall', '-Werror=format-security', '-Wp,-D_FORTIFY_SOURCE=2', '-Wp,-D_GLIBCXX_ASSERTIONS', '-fexceptions', '-fstack-protector-strong', '-grecord-gcc-switches', '-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1', '-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1', '-m64', '-mtune=generic', '-fasynchronous-unwind-tables', '-fstack-clash-protection', '-fcf-protection', '-MMD', '-D_GNU_SOURCE=1', '-D_XOPEN_SOURCE_EXTENDED=1', '-fPIC', '-D__STDC_WANT_LIB_EXT1__=1', '-fstack-protector-strong', '-fstack-clash-protection', '-fvisibility=hidden', '-I.', '-I../..', '-I.', '-I../..', '-Ilib/replace', '-I../../lib/replace', '-DHAVE_PYEXT=1', '-DHAVE_PYTHON_H=1', '../../pytalloc_util.c', '-c', '-o/builddir/build/BUILD/talloc-2.1.16/bin/default/pytalloc_util.c.13.o'] -> task in 'pytalloc-util.objlist' failed with exit status 1: {task 140431097928192: c pytalloc_util.c -> pytalloc_util.c.23.o} ['/usr/bin/gcc', '-O2', '-g', '-pipe', '-Wall', '-Werror=format-security', '-Wp,-D_FORTIFY_SOURCE=2', '-Wp,-D_GLIBCXX_ASSERTIONS', '-fexceptions', '-fstack-protector-strong', '-grecord-gcc-switches', '-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1', '-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1', '-m64', '-mtune=generic', '-fasynchronous-unwind-tables', '-fstack-clash-protection', '-fcf-protection', '-MMD', '-D_GNU_SOURCE=1', '-D_XOPEN_SOURCE_EXTENDED=1', '-fPIC', '-D__STDC_WANT_LIB_EXT1__=1', '-fstack-protector-strong', '-fstack-clash-protection', '-fvisibility=hidden', '-I.', '-I../..', '-I.', '-I../..', '-Ilib/replace', '-I../../lib/replace', '-DPYTHONDIR="/usr/lib/python2.7/site-packages"', '-DPYTHONARCHDIR="/usr/lib64/python2.7/site-packages"', '-DHAVE_PYEXT=1', '-DHAVE_PYTHON_H=1', '../../pytalloc_util.c', '-c', '-o/builddir/build/BUILD/talloc-2.1.16/bin/default/pytalloc_util.c.23.o'] make: *** [Makefile:8: all] Error 1 error: Bad exit status from /var/tmp/rpm-tmp.NyjjQv (%build)
I tried to do some POC changes to use pkg-config instead of python3-config for pyembed e.g. diff --git a/third_party/waf/waflib/Tools/python.py b/third_party/waf/waflib/Tools/python.py index 52a05c668e3..2e80cd4901f 100644 --- a/third_party/waf/waflib/Tools/python.py +++ b/third_party/waf/waflib/Tools/python.py @@ -337,8 +337,9 @@ def check_python_headers(conf, features='pyembed pyext'): xx = env.CXX_NAME and 'cxx' or 'c' if 'pyembed' in features: - for flags in all_flags: - conf.check_cfg(msg='Asking python-config for pyembed %r flags' % ' '.join(flags), path=env.PYTHON_CONFIG, package='', uselib_store='PYEMBED', args=flags) + for flags in [['--cflags', '--libs']]: + conf.check_cfg(msg='Asking pkg-config for pyembed %r flags' % ' '.join(flags), package="python-%s" % env.PYTHON_VERSION, uselib_store='PYEMBED', args=flags) + try: conf.test_pyembed(xx) but -lpython3 was used even for pyext and not just for pyembed [build@host mock]$ grep lpython3 build.log 01:38:42 runner ['/usr/bin/gcc', '-Wl,--version-script=/builddir/build/BUILD/talloc-2.1.16/bin/default/pytalloc-util.cpython-38-x86_64-linux-gnu.vscript', '-shared', '-Wl,-h,libpytalloc-util.cpython-38-x86-64-linux-gnu.so.2', 'pytalloc_util.c.13.o', '-o/builddir/build/BUILD/talloc-2.1.16/bin/default/libpytalloc-util.cpython-38-x86-64-linux-gnu.so', '-Wl,-Bstatic', '-Wl,-Bdynamic', '-L/builddir/build/BUILD/talloc-2.1.16/bin/default', '-L/usr/local/lib', '-ltalloc', '-ldl', '-lcrypt', '-lpython3.8', '-Wl,-z,relro', '-Wl,--as-needed', '-Wl,-z,now', '-specs=/usr/lib/rpm/redhat/redhat-hardened-ld', '-Wl,-no-undefined'] 01:38:42 runner ['/usr/bin/gcc', '-shared', '-fPIC', '-fPIC', 'pytalloc.c.19.o', '-o/builddir/build/BUILD/talloc-2.1.16/bin/default/pytalloc.cpython-38-x86_64-linux-gnu.so', '-Wl,-Bstatic', '-Wl,-Bdynamic', '-L/builddir/build/BUILD/talloc-2.1.16/bin/default', '-L/usr/local/lib', '-L/usr/lib64', '-lpytalloc-util.cpython-38-x86-64-linux-gnu', '-ltalloc', '-lpython3.8', '-lcrypt', '-ldl', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-Wl,-z,relro', '-Wl,--as-needed', '-Wl,-z,now', '-specs=/usr/lib/rpm/redhat/redhat-hardened-ld'] 01:38:42 runner ['/usr/bin/gcc', '-shared', '-fPIC', '-fPIC', 'test_pytalloc.c.21.o', '-o/builddir/build/BUILD/talloc-2.1.16/bin/default/test-pytalloc.cpython-38-x86_64-linux-gnu.so', '-Wl,-Bstatic', '-Wl,-Bdynamic', '-L/builddir/build/BUILD/talloc-2.1.16/bin/default', '-L/usr/local/lib', '-L/usr/lib64', '-lpytalloc-util.cpython-38-x86-64-linux-gnu', '-ltalloc', '-lpython3.8', '-lcrypt', '-ldl', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-Wl,-z,relro', '-Wl,--as-needed', '-Wl,-z,now', '-specs=/usr/lib/rpm/redhat/redhat-hardened-ld'] and -lpython3 should not be used for '-o/builddir/build/BUILD/talloc-2.1.16/bin/default/pytalloc.cpython-38-x86_64-linux-gnu.so' Miro, I would appreciate a help here cause I do not have more time to fight with waf (third_party/waf in samba upstream) and usage of waf in samba(buildtools/wafsamba in samba upstream)
Victor, how do we deal with this breakage? Is there at least some viable workaround?
I've pinged the waf maintainer. We need a proper fix upstream.
Miro: Fix python3-config and everything will be fine.
Andreas: How exactly do I fix it? This is a complex problem: waf uses it both for extension modules and embedded. If we revert the change, we link Python extensions against libpython and that should no longer be done.
(In reply to Lukas Slebodnik from comment #17) > I tried to do some POC changes to use pkg-config instead of python3-config > for pyembed > e.g. > > diff --git a/third_party/waf/waflib/Tools/python.py > b/third_party/waf/waflib/Tools/python.py > index 52a05c668e3..2e80cd4901f 100644 > --- a/third_party/waf/waflib/Tools/python.py > +++ b/third_party/waf/waflib/Tools/python.py > @@ -337,8 +337,9 @@ def check_python_headers(conf, features='pyembed pyext'): > xx = env.CXX_NAME and 'cxx' or 'c' > > if 'pyembed' in features: > - for flags in all_flags: > - conf.check_cfg(msg='Asking python-config for > pyembed %r flags' % ' '.join(flags), path=env.PYTHON_CONFIG, package='', > uselib_store='PYEMBED', args=flags) > + for flags in [['--cflags', '--libs']]: > + conf.check_cfg(msg='Asking pkg-config for > pyembed %r flags' % ' '.join(flags), package="python-%s" % > env.PYTHON_VERSION, uselib_store='PYEMBED', args=flags) > + > > try: > conf.test_pyembed(xx) > Previous patch (using pkg-config for pyembed) works well for libtdb. But it does not work for libtalloc(pytalloc.cpython-38-x86_64-linux-gnu.so) because pytalloc.so has two dependencies libtalloc and libpytalloc-util And it seems that all libs necessary for libpytalloc-util were used also for linking pytalloc.so (autotools/libtool call such feature link_all_deplibs) I am not syre whether it is possible to avoid such behaviour in waf. > > but -lpython3 was used even for pyext and not just for pyembed > > [build@host mock]$ grep lpython3 build.log > 01:38:42 runner ['/usr/bin/gcc', > '-Wl,--version-script=/builddir/build/BUILD/talloc-2.1.16/bin/default/ > pytalloc-util.cpython-38-x86_64-linux-gnu.vscript', '-shared', > '-Wl,-h,libpytalloc-util.cpython-38-x86-64-linux-gnu.so.2', > 'pytalloc_util.c.13.o', > '-o/builddir/build/BUILD/talloc-2.1.16/bin/default/libpytalloc-util.cpython- > 38-x86-64-linux-gnu.so', '-Wl,-Bstatic', '-Wl,-Bdynamic', > '-L/builddir/build/BUILD/talloc-2.1.16/bin/default', '-L/usr/local/lib', > '-ltalloc', '-ldl', '-lcrypt', '-lpython3.8', '-Wl,-z,relro', > '-Wl,--as-needed', '-Wl,-z,now', > '-specs=/usr/lib/rpm/redhat/redhat-hardened-ld', '-Wl,-no-undefined'] > 01:38:42 runner ['/usr/bin/gcc', '-shared', '-fPIC', '-fPIC', > 'pytalloc.c.19.o', > '-o/builddir/build/BUILD/talloc-2.1.16/bin/default/pytalloc.cpython-38- > x86_64-linux-gnu.so', '-Wl,-Bstatic', '-Wl,-Bdynamic', > '-L/builddir/build/BUILD/talloc-2.1.16/bin/default', '-L/usr/local/lib', > '-L/usr/lib64', '-lpytalloc-util.cpython-38-x86-64-linux-gnu', '-ltalloc', > '-lpython3.8', '-lcrypt', '-ldl', '-lcrypt', '-lpthread', '-ldl', '-lutil', ^^^^^^^^^^^^ What will happen if python extension is linked with this library ? > '-lm', '-lm', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', > '-Wl,-z,relro', '-Wl,--as-needed', '-Wl,-z,now', > '-specs=/usr/lib/rpm/redhat/redhat-hardened-ld']
> What will happen if python extension is linked with -lpython3.8? The following 3.8 feature stops working: https://docs.python.org/dev/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build Of course we could probably deliberately make a choice to temporarily ditch the future for libtalloc, libtevent, libldb and samba to make it build, but we should not revert that feature globally (or at least we should not take that decision lightly). Do we know how to make this work for the samba packages at least?
(In reply to Miro Hrončok from comment #23) > > What will happen if python extension is linked with -lpython3.8? > > The following 3.8 feature stops working: > > https://docs.python.org/dev/whatsnew/3.8.html#debug-build-uses-the-same-abi- > as-release-build > I read that section few times and I am not shure what will happen if extension is linked with libpython3.8 But build passed for me including sanity python tests and it works even after installation rpms. <mock-chroot> sh-5.0# python3 Python 3.8.0a4 (default, May 17 2019, 12:33:06) [GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import talloc >>> talloc.total_blocks() 0 >>> talloc.enable_null_tracking() >>> talloc.total_blocks() 1 >>> <mock-chroot> sh-5.0# objdump -p /usr/lib64/python3.8/site-packages/talloc.cpython-38-x86_64-linux-gnu.so | grep NEEDED NEEDED libpytalloc-util.cpython-38-x86-64-linux-gnu.so.2 NEEDED libtalloc.so.2 NEEDED libpython3.8.so.1.0 NEEDED libcrypt.so.2 NEEDED libdl.so.2 NEEDED libpthread.so.0 NEEDED libutil.so.1 NEEDED libm.so.6 NEEDED libc.so.6 > Of course we could probably deliberately make a choice to temporarily ditch > the future for libtalloc, libtevent, libldb and samba to make it build, but > we should not revert that feature globally (or at least we should not take > that decision lightly). > > Do we know how to make this work for the samba packages at least? no idea
It seems like one root issue is that waf "pyembed" fails to detect Python 3.8. Internally, waf creates a C program which runs Py_Initialize(), but the linker fails to find Py_Initialize() because "python3.8-config --libs" output no longer contains -lpython3.8. I reported the issue to waf: https://gitlab.com/ita1024/waf/issues/2239 Fixing waf & other programs requires to add a new --embd option to python3.8-config and pkg-config python3.8 to include -lpython3.8. I'm working on implementing https://bugs.python.org/issue36721
First we need to fix waf, then import the new waf version into Samba. Once we have that, we can fix wafsamba.
*** Bug 1712602 has been marked as a duplicate of this bug. ***
Created attachment 1572856 [details] Patch for all libtdb, libtalloc, libtevent, libldb and samba Assigned a patch that applies to all the affected packages. Running Copr builds of all of them, got as far as libldb for now. Submitted upstream: https://gitlab.com/ita1024/waf/merge_requests/2236
> Assigned a patch Attached
(In reply to Miro Hrončok from comment #29) > > Assigned a patch > > Attached Thank you very much. It works in similar way as POC version with using pkg-config (comment 17) But python extension is still linked with -lpython3.8 [build@host mock]$ grep lpython3 build.log 16:20:06 runner ['/usr/bin/gcc', '-Wl,--version-script=/builddir/build/BUILD/talloc-2.1.16/bin/default/pytalloc-util.cpython-38-x86_64-linux-gnu.vscript', '-shared', '-fPIC', '-fPIC', '-Wl,-h,libpytalloc-util.cpython-38-x86-64-linux-gnu.so.2', 'pytalloc_util.c.13.o', '-o/builddir/build/BUILD/talloc-2.1.16/bin/default/libpytalloc-util.cpython-38-x86-64-linux-gnu.so', '-Wl,-Bstatic', '-Wl,-Bdynamic', '-L/builddir/build/BUILD/talloc-2.1.16/bin/default', '-L/usr/local/lib', '-ltalloc', '-ldl', '-lcrypt', '-lpython3.8', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-lpython3.8', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-Wl,-z,relro', '-Wl,--as-needed', '-Wl,-z,now', '-specs=/usr/lib/rpm/redhat/redhat-hardened-ld', '-Wl,-no-undefined'] 16:20:06 runner ['/usr/bin/gcc', '-shared', '-fPIC', '-fPIC', '-fPIC', '-fPIC', 'pytalloc.c.19.o', '-o/builddir/build/BUILD/talloc-2.1.16/bin/default/pytalloc.cpython-38-x86_64-linux-gnu.so', '-Wl,-Bstatic', '-Wl,-Bdynamic', '-L/builddir/build/BUILD/talloc-2.1.16/bin/default', '-L/usr/local/lib', '-L/usr/lib64', '-L/usr/lib64', '-lpytalloc-util.cpython-38-x86-64-linux-gnu', '-ltalloc', '-lpython3.8', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-lpython3.8', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-lcrypt', '-ldl', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-Wl,-z,relro', '-Wl,--as-needed', '-Wl,-z,now', '-specs=/usr/lib/rpm/redhat/redhat-hardened-ld'] 16:20:06 runner ['/usr/bin/gcc', '-shared', '-fPIC', '-fPIC', '-fPIC', '-fPIC', 'test_pytalloc.c.21.o', '-o/builddir/build/BUILD/talloc-2.1.16/bin/default/test-pytalloc.cpython-38-x86_64-linux-gnu.so', '-Wl,-Bstatic', '-Wl,-Bdynamic', '-L/builddir/build/BUILD/talloc-2.1.16/bin/default', '-L/usr/local/lib', '-L/usr/lib64', '-L/usr/lib64', '-lpytalloc-util.cpython-38-x86-64-linux-gnu', '-ltalloc', '-lpython3.8', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-lpython3.8', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-lcrypt', '-ldl', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lm', '-Wl,-z,relro', '-Wl,--as-needed', '-Wl,-z,now', '-specs=/usr/lib/rpm/redhat/redhat-hardened-ld'] <mock-chroot> sh-5.0# objdump -p /builddir/build/BUILD/talloc-2.1.16/bin/default/pytalloc.cpython-38-x86_64-linux-gnu.so | grep NEEDED NEEDED libpytalloc-util.cpython-38-x86-64-linux-gnu.so.2 NEEDED libtalloc.so.2 NEEDED libpython3.8.so.1.0 NEEDED libcrypt.so.2 NEEDED libpthread.so.0 NEEDED libdl.so.2 NEEDED libutil.so.1 NEEDED libm.so.6 NEEDED libc.so.6 It seems to be caused by linking extensions with all dependencies of libpytalloc-util.so (as I already mentioned in comment 22) I will be fine with applying patch in fedora if it does not cause any problem for python3.8 (BTW python test passed in %check phase)
IMHO linking the extensions against libpython is not a very severe issue.
(In reply to Miro Hrončok from comment #31) > IMHO linking the extensions against libpython is not a very severe issue. Great Victor, can you confirm the same ? I'll rather double check.
Linking an extension to libpython is perfectly safe. Python always did that for all C extensions on all platforms. Only Python 3.8 started to no longer link C extensions to libpython on some platforms: Unix except Android and Cygwin. https://docs.python.org/dev/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build libtalloc and libtdb can be linked to libpython. The main drawback is that you loose the ability to use a debug build of Python with libtalloc and libtdb compiled in release mode. It's a minor issue ;-)
(In reply to Victor Stinner from comment #33) > Linking an extension to libpython is perfectly safe. Python always did that > for all C extensions on all platforms. Only Python 3.8 started to no longer > link C extensions to libpython on some platforms: Unix except Android and > Cygwin. > https://docs.python.org/dev/whatsnew/3.8.html#debug-build-uses-the-same-abi- > as-release-build > > libtalloc and libtdb can be linked to libpython. > > The main drawback is that you loose the ability to use a debug build of > Python with libtalloc and libtdb compiled in release mode. It's a minor > issue ;-) Just libtalloc and libldb will be linked with lpython3.8 (due to nested lining done in waf libtevent and libtdb are fine.
I hope the patch is now considered OK. let me know if I shall send a PR, commit directly or if you prefer to this yourself.
(In reply to Miro Hrončok from comment #35) > I hope the patch is now considered OK. let me know if I shall send a PR, > commit directly or if you prefer to this yourself. Patches+builds are already in rawhide for lib{talloc, tevent, tdb, ldb} Andreas, could you add patch to samba dist-git ? https://bugzilla.redhat.com/attachment.cgi?id=1572856
> Andreas, could you add patch to samba dist-git ? > https://bugzilla.redhat.com/attachment.cgi?id=1572856 FYI Miro made this change upstream in waf and waf 2.0.17 has been released with the fix.
waf update MR for Samba: https://gitlab.com/samba-team/samba/merge_requests/519 Once that is in I will backport to the 4.10 branch. We have to do new releases of Samba and the libs then.
(In reply to Andreas Schneider from comment #38) > waf update MR for Samba: > https://gitlab.com/samba-team/samba/merge_requests/519 > > Once that is in I will backport to the 4.10 branch. We have to do new > releases of Samba and the libs then. Have you consider patching it in dist-git meanwhile? IMHO we needn't wait for next regular 4.10.x release
See also bz1718113.
This issue is blocking the Python 3.8 rebuilds. If this package won't build with 3.8, it won't be installable, along with all its dependent packages, after the side tag is merged. Furthermore, as it fails to install, its dependent packages will fail to install and/or build as well. The fix should be pushed on the master branch and no release bump is required. Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.8: https://copr.fedorainfracloud.org/coprs/g/python/python3.8/ This issue needs to be resolved by next week, if other rebuilds of Python 3.8 beta 1 go well. If this is unrealistic for you, let us know how much time you need. If you don't have free cycles to dedicate fixing your package, notify us and we'll try to provide some pointers. Let us know if we can push a fix directly without a pull request, in the case we happen to have one before you do. We recommend always consulting with upstream, chances are this is already fixed there.
This issue is blocking the Python 3.8 rebuilds. This package is critical for Fedora, unless this is resolved, we cannot proceed with the side tag merge.
https://bugzilla.samba.org/show_bug.cgi?id=13960
(In reply to Lukas Slebodnik from comment #39) > Have you consider patching it in dist-git meanwhile? > IMHO we needn't wait for next regular 4.10.x release
With the above patch you can patch samba and do it. However there is also bug #1718113 which needs to be fixed. Waiting for upstream to accept it.
(In reply to Andreas Schneider from comment #45) > With the above patch you can patch samba and do it. However there is also > bug #1718113 which needs to be fixed. Waiting for upstream to accept it. Issue with "time.clock()" in BZ#1718113 is not critical. Because it is used just in debug messages and does not have any significant impact on buildsystem. So even removing lines with problematic debug messages would fix the problem. Do you really want to wait for upstream?
(In reply to Andreas Schneider from comment #45) > With the above patch you can patch samba and do it. However there is also > bug #1718113 which needs to be fixed. Waiting for upstream to accept it. Patch for BZ#1718113 is already in upstream https://gitlab.com/samba-team/samba/merge_requests/546 Is there any blocker to backport patches to rawhide?
(In reply to Lukas Slebodnik from comment #47) > Is there any blocker to backport patches to rawhide? There was, however Python 3.8 will most likely be deferred to Fedora 32, so there is not right now. That said, having the patch applied in master will make our work significantly easier.
Can we please get this fixed?
Pushed as 1ac5f2f4b603b67eefbde338c9d96d633ad4b96d