Bug 1420086

Summary: libboost_python3.so.1.63.0 uses Python 2 not Python 3
Product: [Fedora] Fedora Reporter: Antonio T. (sagitter) <anto.trande>
Component: boostAssignee: Jonathan Wakely <jwakely>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: anto.trande, dakingun, denis.arnaud_fedora, jwakely, kevin, mattia.verga, me, stefan
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-02-16 22:20:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Antonio T. (sagitter) 2017-02-07 18:25:49 UTC
Description of problem:
'boost' is required by many packages.
Must be rebuilt with GCC7 ? 

Version-Release number of selected component (if applicable):
1.63.0

Comment 1 Jonathan Wakely 2017-02-07 20:20:46 UTC
No, boost has already been rebuilt. The problem is probably in other packages, but your bug report is so vague it's impossible to tell.

What is the actual problem you see? What did you do to see the problem?

Comment 2 Antonio T. (sagitter) 2017-02-08 10:44:28 UTC
I asked to myself why 'boost' has not been rebuilt yet with gcc7.
I found this unexpected problem with 'antimony': https://github.com/mkeeter/antimony/issues/178#issuecomment-277997765

Comment 3 Jonathan Wakely 2017-02-08 12:54:10 UTC
The boost-1.63.0-2 build did miss gcc 7 landing in the buildroot by a few hours, but I've tried rebuilding it and it doesn't affect that symbol, which is already present in rawhide:

$ mock -r fedora-rawhide-x86_64 -q  --install boost-python3 binutils
$ mock -r fedora-rawhide-x86_64 -q --copyout /usr/lib64/libboost_python3.so.1.63.0 /tmp/libboost_python3.so.1.63.0
$ nm -D -C /tmp/libboost_python3.so.1.63.0 | fgrep init_module
000000000002fcc0 T boost::python::detail::init_module(char const*, void (*)())

Comment 4 Jonathan Wakely 2017-02-08 15:25:36 UTC
You could try manually installing the RPMs from this scratch build:
https://koji.fedoraproject.org/koji/taskinfo?taskID=17675153

Do you still get the undefined references with that?

Comment 5 Antonio T. (sagitter) 2017-02-08 16:29:25 UTC
> Do you still get the undefined references with that?

Yes, unfortunately; just on rawhide. And yet it looks correctly linked:

/usr/bin/c++   -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -fPIC -pie -DNDEBUG -DRELEASE  -Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -fPIC -pie CMakeFiles/antimony.dir/app/main.cpp.o CMakeFiles/antimony.dir/app/app.cpp.o CMakeFiles/antimony.dir/app/update.cpp.o CMakeFiles/antimony.dir/app/colors.cpp.o CMakeFiles/antimony.dir/canvas/scene.cpp.o CMakeFiles/antimony.dir/canvas/canvas_view.cpp.o CMakeFiles/antimony.dir/canvas/info.cpp.o CMakeFiles/antimony.dir/canvas/inspector/frame.cpp.o CMakeFiles/antimony.dir/canvas/inspector/export.cpp.o CMakeFiles/antimony.dir/canvas/inspector/title.cpp.o CMakeFiles/antimony.dir/canvas/inspector/buttons.cpp.o CMakeFiles/antimony.dir/canvas/inspector/util.cpp.o CMakeFiles/antimony.dir/canvas/datum_row.cpp.o CMakeFiles/antimony.dir/canvas/datum_editor.cpp.o CMakeFiles/antimony.dir/canvas/datum_port.cpp.o CMakeFiles/antimony.dir/canvas/subdatum/subdatum_frame.cpp.o CMakeFiles/antimony.dir/canvas/subdatum/subdatum_editor.cpp.o CMakeFiles/antimony.dir/canvas/subdatum/subdatum_row.cpp.o CMakeFiles/antimony.dir/canvas/connection/base.cpp.o CMakeFiles/antimony.dir/canvas/connection/connection.cpp.o CMakeFiles/antimony.dir/canvas/connection/dummy.cpp.o CMakeFiles/antimony.dir/window/base.cpp.o CMakeFiles/antimony.dir/window/canvas.cpp.o CMakeFiles/antimony.dir/window/viewport.cpp.o CMakeFiles/antimony.dir/window/quad.cpp.o CMakeFiles/antimony.dir/window/base_viewport_window.cpp.o CMakeFiles/antimony.dir/window/script_window.cpp.o CMakeFiles/antimony.dir/graph/constructor/populate.cpp.o CMakeFiles/antimony.dir/graph/proxy/graph.cpp.o CMakeFiles/antimony.dir/graph/proxy/node.cpp.o CMakeFiles/antimony.dir/graph/proxy/script.cpp.o CMakeFiles/antimony.dir/graph/proxy/datum.cpp.o CMakeFiles/antimony.dir/graph/proxy/subdatum.cpp.o CMakeFiles/antimony.dir/graph/proxy/base_datum.cpp.o CMakeFiles/antimony.dir/graph/hooks/hooks.cpp.o CMakeFiles/antimony.dir/graph/hooks/export.cpp.o CMakeFiles/antimony.dir/graph/hooks/ui.cpp.o CMakeFiles/antimony.dir/graph/hooks/title.cpp.o CMakeFiles/antimony.dir/graph/serialize/serializer.cpp.o CMakeFiles/antimony.dir/graph/serialize/deserializer.cpp.o CMakeFiles/antimony.dir/script/syntax.cpp.o CMakeFiles/antimony.dir/script/editor.cpp.o CMakeFiles/antimony.dir/script/frame.cpp.o CMakeFiles/antimony.dir/undo/undo_command.cpp.o CMakeFiles/antimony.dir/undo/undo_add_node.cpp.o CMakeFiles/antimony.dir/undo/undo_add_datum.cpp.o CMakeFiles/antimony.dir/undo/undo_delete_datum.cpp.o CMakeFiles/antimony.dir/undo/undo_delete_multi.cpp.o CMakeFiles/antimony.dir/undo/undo_delete_node.cpp.o CMakeFiles/antimony.dir/undo/undo_delete_link.cpp.o CMakeFiles/antimony.dir/undo/undo_move_node.cpp.o CMakeFiles/antimony.dir/undo/undo_move_datum.cpp.o CMakeFiles/antimony.dir/undo/undo_change_script.cpp.o CMakeFiles/antimony.dir/undo/undo_change_expr.cpp.o CMakeFiles/antimony.dir/undo/undo_stack.cpp.o CMakeFiles/antimony.dir/dialog/exporting.cpp.o CMakeFiles/antimony.dir/dialog/resolution.cpp.o CMakeFiles/antimony.dir/export/export_mesh.cpp.o CMakeFiles/antimony.dir/export/export_heightmap.cpp.o CMakeFiles/antimony.dir/export/export_worker.cpp.o CMakeFiles/antimony.dir/viewport/scene.cpp.o CMakeFiles/antimony.dir/viewport/gl.cpp.o CMakeFiles/antimony.dir/viewport/view.cpp.o CMakeFiles/antimony.dir/viewport/image.cpp.o CMakeFiles/antimony.dir/viewport/control/control.cpp.o CMakeFiles/antimony.dir/viewport/control/control_instance.cpp.o CMakeFiles/antimony.dir/viewport/control/point.cpp.o CMakeFiles/antimony.dir/viewport/control/wireframe.cpp.o CMakeFiles/antimony.dir/viewport/render/instance.cpp.o CMakeFiles/antimony.dir/viewport/render/task.cpp.o CMakeFiles/antimony.dir/antimony_automoc.cpp.o CMakeFiles/antimony.dir/antimony_automoc.dir/qrc_gl_2HPAS2ZRQXYTL6.cpp.o  -o antimony -rdynamic /usr/lib64/libQt5OpenGL.so.5.7.1 /usr/lib64/libQt5Network.so.5.7.1 /usr/lib64/libQt5Concurrent.so.5.7.1 ../lib/graph/libSbGraph.a ../lib/fab/libSbFab.a /usr/lib64/libQt5Widgets.so.5.7.1 /usr/lib64/libQt5Gui.so.5.7.1 /usr/lib64/libQt5Core.so.5.7.1 -lpython3.6m -lboost_python3 -lpng -lz

Comment 6 Jonathan Wakely 2017-02-08 16:51:15 UTC
So it's not a gcc6 vs gcc7 problem then.

Comment 7 Antonio T. (sagitter) 2017-02-08 19:52:14 UTC
I don't know.
The same src-rpm is successfully compiled on fedora 25 but not on rawhide.

Comment 8 Jonathan Wakely 2017-02-09 03:29:36 UTC
(In reply to Jonathan Wakely from comment #3)
> The boost-1.63.0-2 build did miss gcc 7 landing in the buildroot by a few
> hours, but I've tried rebuilding it and it doesn't affect that symbol, which
> is already present in rawhide:
> 
> $ mock -r fedora-rawhide-x86_64 -q  --install boost-python3 binutils
> $ mock -r fedora-rawhide-x86_64 -q --copyout
> /usr/lib64/libboost_python3.so.1.63.0 /tmp/libboost_python3.so.1.63.0
> $ nm -D -C /tmp/libboost_python3.so.1.63.0 | fgrep init_module
> 000000000002fcc0 T boost::python::detail::init_module(char const*, void
> (*)())

Oops! This isn't the same symbol at all. The undefined reference is to boost::python::detail::init_module(PyModuleDef&, void (*)()) 

But both libboost_python.so and libboost_python3.so define the same symbol:

boost::python::detail::init_module(char const*, void (*)())

That's only meant to be used for Python2.

And the libboost_python3.so library is linked to the wrong Python lib:

<mock-chroot> sh-4.4# ldd /usr/lib64/libboost_python3.so | fgrep python
        libpython2.7.so.1.0 => /lib64/libpython2.7.so.1.0 (0x00007f5160477000)

So something is wrong with the boost build.

Comment 9 Antonio T. (sagitter) 2017-02-12 16:40:14 UTC
>So something is wrong with the boost build.

Please, let me know when it's fixed.

Comment 10 Mattia Verga 2017-02-12 17:52:38 UTC
May be related to https://svn.boost.org/trac/boost/ticket/12515

Comment 11 Stefan Seefeld 2017-02-13 14:30:56 UTC
What is the exact command used to build Boost.Python in this context (which I may use to reproduce the mangled Py2/Py3 dependency) ?

Comment 12 Jonathan Wakely 2017-02-13 14:41:48 UTC
The boost.spec file used for building the RPMs does this:

export 'RPM_OPT_FLAGS=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -fno-strict-aliasing -Wno-unused-local-typedefs -Wno-deprecated-declarations'

cat > ./tools/build/src/user-config.jam << "EOF"
import os ;
local RPM_OPT_FLAGS = [ os.environ RPM_OPT_FLAGS ] ;

using gcc : : : <compileflags>$(RPM_OPT_FLAGS) ;
using python : 2.7 : /usr/bin/python2 : /usr/include/python2.7 : : : : ;
using python : 3.6 : /usr/bin/python3 : /usr/include/python3.6m : : : : m ;
%endif
EOF

./bootstrap.sh --with-toolset=gcc --with-icu

# The "python=2.*" bit tells jam that we want to _also_ build 2.*, not
# just 3.*.  When omitted, it just builds for python 3 twice, once
# calling the library libboost_python and once libboost_python3.  I
# assume this is for backward compatibility for apps that are used to
# linking against -lboost_python, for when 2->3 transition is
# eventually done.

./b2 -d+2 -q -j6 --without-mpi --without-graph_parallel --build-dir=serial variant=release threading=multi debug-symbols=on pch=off python=2.7 stage

Comment 13 Stefan Seefeld 2017-02-13 16:01:28 UTC
I'm pretty sure that right now it is not possible to build Boost.Python for both Python2 and Python3 with a single build command. Thus I would advise to instead do:

1) generate user-config.jam with "using python : 2.7 ..." in it.
2) invoke `b2`
3) package up the built libraries
4) generate user-config.jam with "using python : 3.x ..." in it.
5) invoke `b2`
6) package up the built libraries

Note that this should only affect libraries using Python, such as Boost.Python as well as the Boost.MPI Python bindings. All other Boost libraries should be built identically, and thus don't need to be built twice. So if you insert a "cleanup" step after 3) that removes only the libraries dependent on Python rather than the entire build tree, 5) should only rebuild those libraries.

(There may be a way to "fix" b2 to allow building for multiple Python versions in a single run, but I have always upheld that this is a feature with little use adding lots of complications to the build logic. Just running one build per desired variant seemed far simpler in every respect...)

Comment 14 Kevin Kofler 2017-02-13 20:10:28 UTC
Can't we just revert the commit that was pinpointed as the source of the regression:
https://github.com/boostorg/build/commit/78ffbe094400d277627b2c19ceb182d637b8baca
in a Fedora patch?

Comment 15 Stefan Seefeld 2017-02-13 20:24:23 UTC
Note that I have sent a mail to the Boost.Build maintainers as even I don't understand what the expected / desired behaviour is. My suggestion above is merely a means to get things working quickly without any need to modify upstream code.

As far as the Boost.Python build logic is concerned, I don't think building two variants in a single run would work, even with your commit reversal (though I'd be happy to be proven wrong). As I said, I'd rather invoke `b2` twice to build two distinct variants rather than having to worry about all the involved complexity of avoiding multiple build artefacts colliding due to mixing variants.

Comment 16 Jonathan Wakely 2017-02-16 19:57:23 UTC
The problem is fixed in boost-1.63.0-4.fc26 which is building now.