Bug 1217376 - distutils embed "hardening" compiler/linker flags
Summary: distutils embed "hardening" compiler/linker flags
Keywords:
Status: CLOSED DUPLICATE of bug 1668029
Alias: None
Product: Fedora
Classification: Fedora
Component: python3
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Miro Hrončok
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1514358 (view as bug list)
Depends On: 1218294 1543394
Blocks: Fedora28BuildFlags 1548822 1548823
TreeView+ depends on / blocked
 
Reported: 2015-04-30 09:09 UTC by Dan Horák
Modified: 2019-01-21 22:19 UTC (History)
30 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-01-21 22:19:54 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Dan Horák 2015-04-30 09:09:10 UTC
distutils embed the "hardening" compiler/linker flags taken from Fedora global flags. Using this python tool for compiling stuff is not possible without installing redhat-rpm-config that provides the hardening configuration for the compiler and linker. Using distutils shouldn't enforce the use of redhat-rpm-config. And also hardening doesn't work for all applications and can be disabled on package level.

from lvm build bot logs:
...
make -C python
make[1]: Entering directory '/srv/buildbot/lvm2-slave/Fedora_Rawhide_x86_64_KVM/build/python'
ln -s ./liblvm.c liblvm_python.c
/usr/bin/python setup.py build
running build
running build_ext
building 'lvm' extension
creating build
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I../include -I/usr/include/python2.7 -c liblvm_python.c -o build/temp.linux-x86_64-2.7/liblvm_python.o
gcc: error: /usr/lib/rpm/redhat/redhat-hardened-cc1: No such file or directory
error: command 'gcc' failed with exit status 1
Makefile:23: recipe for target '.liblvm_built' failed
make[1]: *** [.liblvm_built] Error 1
rm liblvm_python.c
make[1]: Leaving directory '/srv/buildbot/lvm2-slave/Fedora_Rawhide_x86_64_KVM/build/python'
make.tmpl:310: recipe for target 'python' failed
make: *** [python] Error 2


looking at the content of python-libs-2.7.9-10.fc23.x86_64.rpm shows
[dan@eagle p]$ grepfind hardened
./python2.7/config/Makefile:70:OPT=		-DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv
./python2.7/config/Makefile:72:CFLAGS=		$(BASECFLAGS) -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv  $(OPT) $(EXTRA_CFLAGS)
./python2.7/config/Makefile:77:LDFLAGS=	-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld 
./python2.7/config/Makefile:164:CONFIG_ARGS=	 '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--enable-ipv6' '--enable-shared' '--enable-unicode=ucs4' '--with-dbmliborder=gdbm:ndbm:bdb' '--with-system-expat' '--with-system-ffi' '--with-dtrace' '--with-tapset-install-dir=/usr/share/systemtap/tapset' '--with-valgrind' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CC=gcc' 'CFLAGS=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv ' 'LDFLAGS=-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld ' 'CPPFLAGS='
Binary file ./python2.7/_sysconfigdata.pyc matches
Binary file ./python2.7/_sysconfigdata.pyo matches
./python2.7/_sysconfigdata.py:20: 'BLDSHARED': 'gcc -pthread -shared -Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld',
./python2.7/_sysconfigdata.py:26: 'CFLAGS': '-fno-strict-aliasing -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv  -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv',
./python2.7/_sysconfigdata.py:29: 'CONFIG_ARGS': "'--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--enable-ipv6' '--enable-shared' '--enable-unicode=ucs4' '--with-dbmliborder=gdbm:ndbm:bdb' '--with-system-expat' '--with-system-ffi' '--with-dtrace' '--with-tapset-install-dir=/usr/share/systemtap/tapset' '--with-valgrind' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CC=gcc' 'CFLAGS=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv ' 'LDFLAGS=-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld ' 'CPPFLAGS='",
./python2.7/_sysconfigdata.py:372: 'LDFLAGS': '-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld',
./python2.7/_sysconfigdata.py:376: 'LDSHARED': 'gcc -pthread -shared -Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld',
./python2.7/_sysconfigdata.py:416: 'OPT': '-DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv',
./python2.7/_sysconfigdata.py:449: 'PY_CFLAGS': '-fno-strict-aliasing -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv  -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -I. -IInclude -I/builddir/build/BUILD/Python-2.7.9/Include -fPIC -DPy_BUILD_CORE',


Version-Release number of selected component (if applicable):
python-2.7.9-10.fc23

Comment 1 Thomas Spura 2015-04-30 09:21:17 UTC
I think python should use the hardening flags by default. Would you suggest to grep them from redhat-rpm-config on buildtime so that distutils works without requiring redhat-rpm-config?
Just requiring redhat-rpm-config would be the easiest.

Comment 2 Dan Horák 2015-04-30 09:45:12 UTC
Because the hardening is optional part of the distro flags (although enabled by default), then in my opinion distutils shouldn't enforce it. Ideally it should evaluate the flags during runtime (eg. by doing "rpm --eval %__global_cflags/%__global_ldflags"). The current situation breaks upstream development on Fedora that doesn't want to use the Fedora default flags (redhat-rpm-config not installed).

Comment 3 Bohuslav "Slavek" Kabrda 2015-04-30 13:08:23 UTC
So here's why this happens:
During build of Python itself, the _sysconfigdata.py file is created. It contains all sorts of values from build configuration of Python build - paths, soname, etc. Among others, it also saves the compilation flags.
Distutils assume that these flags represent the "best" way to build a C package on the system (IIUC), so they use them as default.

This makes me wonder what actually is the best solution here. As Thomas mentioned, we could just add dependency on redhat-rpm-config and be done with it. OTOH it's true that not everything should be/needs to be hardened.

Taking this more generally, we could probably argue about all flags that are here, e.g. should distutils use -fpic? Should distutils use -fwrapv? And so on. I would be very hesitant about changing these without approval of FPC. If you really think this should be changed, I suggest we open an FPC ticket. My opinion is that we should use what Tom suggested - depending on redhat-rpm-config.

Comment 4 Nick Coghlan 2015-05-04 04:07:29 UTC
+1 for depending on redhat-rpm-config as an interim solution, as the default flags indicate our best recommendation for how to build C/C++ software for our platforms.

Longer term, it may make sense to review the individual flags to see if there should be a delta between the default flags used for Python extension modules and those used for other C/C++ software.

Comment 5 Bohuslav "Slavek" Kabrda 2015-05-04 14:17:49 UTC
There's a small issue (or rather inconvenience) here - if I add redhat-rpm-config as dependency of python-devel (*), redhat-rpm-config will also draw along bunch of unnecessary deps, such as perl-srpm-macros etc. Therefore I requested that /usr/lib/rpm/redhat/redhat-hardened-cc1 and /usr/lib/rpm/redhat/redhat-hardened-ld get separated into a subpackage that wouldn't depend on these unnecessary (from Python's POV) packages. I'll wait untill this is done and then add dependency on this new package to python-devel (and also python3-devel).

(*) I believe that this dependency should be added to python-devel and not to python-libs. Although python-libs contain the values referencing redhat-rpm-macros, users can't really compile dependencies without Python headers, which are in python-devel. So I think it's best to keep deps of python-libs (which gets installed pretty much everywhere) minimal and rather leave this dependency to python-devel (which is only installed by people who actually want to build C extensions).

Comment 6 Nick Coghlan 2015-05-06 07:00:50 UTC
Slavek's suggested path forward sounds good to me (including having the new dependency be from python-devel rather than python-libs).

Comment 7 Fedora Admin XMLRPC Client 2015-05-12 12:02:07 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 8 Jan Kurik 2015-07-15 14:13:19 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 23 development cycle.
Changing version to '23'.

(As we did not run this process for some time, it could affect also pre-Fedora 23 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 23 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora23

Comment 9 Lenz Grimmer 2015-12-09 17:50:11 UTC
Seems like this is still an open issue then? I just ran into this while creating a Python virtualenv for installing nikola. The build aborts with the following error:

    creating build/temp.linux-x86_64-2.7/bundled/libsodium/src/libsodium/crypto_aead/chacha20poly1305/sodium
    
    gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DNATIVE_LITTLE_ENDIAN=1 -Ibundled/libsodium/src/libsodium/include -Ibundled/libsodium/src/libsodium/include/sodium -I/usr/include/python2.7 -c buildutils/initlibsodium.c -o build/temp.linux-x86_64-2.7/buildutils/initlibsodium.o
    
    gcc: error: /usr/lib/rpm/redhat/redhat-hardened-cc1: No such file or directory
    
    error: command 'gcc' failed with exit status 1
    
    ----------------------------------------
    Command "/home/lenz/virtualenv/nikola/bin/python2 -c "import setuptools, tokenize;__file__='/tmp/pip-build-HDAJlN/pyzmq/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-n7xR4s-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/lenz/virtualenv/nikola/include/site/python2.7" failed with error code 1 in /tmp/pip-build-HDAJlN/pyzmq

Comment 10 Miguel Morales 2016-01-09 14:08:32 UTC
I got to this same issue "gcc: error: /usr/lib/rpm/redhat/redhat-hardened-cc1: No such file or directory" trying to install the jupyter notebook using pip. Thus, I'm not sure that my solution applies to people that post on this thread, but I thought to create an account a leave a comment on what solved this problem in my case and see if it helps anybody else.


This solved it for me:

sudo dnf install redhat-rpm-config



After that the gcc error went away.

Comment 11 Robert Kuska 2016-01-14 09:20:35 UTC
It's mentioned in comments above, proper way to fix this issue is to have python(3)-devel depending on redhat-rpm-config, the problem is that redhat-rpm-config currently installs bunch of unneeded[*] packages so it was put on hold until rhbz#1218294 is resolved.

[*]
> $ rpm -q redhat-rpm-config --requires                                        
/bin/bash
/bin/sh
coreutils
dwz >= 0.4
ghc-srpm-macros
gnat-srpm-macros
go-srpm-macros
ocaml-srpm-macros
perl-srpm-macros
rpm >= 4.11.0
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(FileDigests) <= 4.6.0-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(PayloadIsXz) <= 5.2-1
zip

Comment 12 Fedora Admin XMLRPC Client 2016-01-29 13:05:47 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 13 Fedora End Of Life 2016-11-24 11:45:00 UTC
This message is a reminder that Fedora 23 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 23. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '23'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 23 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 14 Fedora Admin XMLRPC Client 2017-01-10 18:48:34 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 15 Petr Viktorin (pviktori) 2017-08-08 15:20:47 UTC
Quoting Panu Matilainen from bug #1218294:

> The foo-srpm-macros packages are really small and should
> not bring in further dependencies (that'd be bug in them
> basically). This seems like pointless complication of things,
> redhat-rpm-config is basically expected to be installed
> just about everywhere.

I tend to agree; python[23]-devel should just BuildRequire redhat-rpm-config.

Comment 16 Miro Hrončok 2017-11-09 11:42:17 UTC
python3 merged in https://src.fedoraproject.org/rpms/python3/pull-request/15

Comment 17 Miro Hrončok 2017-11-09 12:03:13 UTC
python2 requested at https://src.fedoraproject.org/rpms/python2/pull-request/6

Comment 18 Mamoru TASAKA 2017-11-20 06:46:44 UTC
*** Bug 1514358 has been marked as a duplicate of this bug. ***

Comment 19 Florian Weimer 2017-11-20 08:57:00 UTC
(In reply to Petr Viktorin from comment #15)
> Quoting Panu Matilainen from bug #1218294:
> 
> > The foo-srpm-macros packages are really small and should
> > not bring in further dependencies (that'd be bug in them
> > basically). This seems like pointless complication of things,
> > redhat-rpm-config is basically expected to be installed
> > just about everywhere.
> 
> I tend to agree; python[23]-devel should just BuildRequire redhat-rpm-config.

I think this is not the right fix.  You really need to stop copying the compiler flags.  I do not see the point of this bizarre behavior.  How does this help our users?

Comment 20 Petr Viktorin (pviktori) 2017-11-20 09:46:03 UTC
(In reply to Florian Weimer from comment #19)
> I think this is not the right fix.  You really need to stop copying the
> compiler flags.  I do not see the point of this bizarre behavior.  How does
> this help our users?

The reason for copying the flags is to make the modules built with distutils compatible with the Python binaries.

What set of compiler flags do you think distutils should use instead?

Comment 21 Florian Weimer 2017-11-20 12:29:45 UTC
(In reply to Petr Viktorin from comment #20)
> (In reply to Florian Weimer from comment #19)
> > I think this is not the right fix.  You really need to stop copying the
> > compiler flags.  I do not see the point of this bizarre behavior.  How does
> > this help our users?
> 
> The reason for copying the flags is to make the modules built with distutils
> compatible with the Python binaries.

Ahh.  The flags injected into RPM builds are carefully tuned *not* to change ABI, so that you can compile software without these hardening flags if you want.  In fact, we would *never* change ABI in this way.

> What set of compiler flags do you think distutils should use instead?

Ideally, it would look at CC, CFLAGS, CXXFLAGS in the environment when distutils is run and use that.  So do the same thing which it currently does at build time, but at run time, with the current process environment.

Comment 22 Petr Viktorin (pviktori) 2017-11-27 12:42:57 UTC
Looking through the CPython build system, in Python 3 we can add Fedora's default flags to CFLAGS_NODIST instead of CFLAGS. That way they won't end up baked in distutils.

Comment 23 Petr Viktorin (pviktori) 2018-01-25 13:28:30 UTC
For Python 3, we need to look at the default flags again and decide which of them should be in CFLAGS_NODIST (ones that don't propagate to distutil-built extensions, e.g. warning settings) and which should be in CFLAGS (e.g. hardening).

Realistically, I don't see this getting fixed for Python 2.

Comment 24 Florian Weimer 2018-01-25 13:42:05 UTC
(In reply to Petr Viktorin from comment #23)
> For Python 3, we need to look at the default flags again and decide which of
> them should be in CFLAGS_NODIST (ones that don't propagate to distutil-built
> extensions, e.g. warning settings) and which should be in CFLAGS (e.g.
> hardening).

I don't quite understand the terminology.  Ideally, you would get everything you need except for the activation of preprocessor and linker flags from CFLAGS, CXXFLAGS, and LDFLAGS.  At least if these variables are set.

> Realistically, I don't see this getting fixed for Python 2.

We need to fix this mostly for Python 3, so reassigning.

Comment 25 Petr Viktorin (pviktori) 2018-01-31 09:51:12 UTC
I don't expect users to set their CFLAGS/LDFLAGS properly when they build extensions. The expectation is that distutils gives you good options rather than configurability.

Currently, on Fedora, the $RPM_OPT_FLAGS are:
-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection

And we add additional flags in the specfile:
-D_GNU_SOURCE -fPIC -fwrapv


We need to determine which of these should be used for *only* building RPMs, and which ones should be passed for user compilation as well.

I understand the above correctly, you are suggesting that:
- the $RPM_OPT_FLAGS should only be used for building Python, not extensions, and
- We instruct setuptools to only use "-O2 -Wall -g -D_GNU_SOURCE -fPIC -fwrapv" for building extensions.

But what about things like -fexceptions -m64 -grecord-gcc-switches? Wouldn't it be good to use these for the extensions as well?
I understand it doesn't change the ABI. But I'm afraid users would lose warnings/optimizations/debug info. Is that just irrational fear?

(I'm probably sounding ignorant here. I still haven't researched the topic as much as I'd like to; I figure asking you will be faster.)

Comment 26 Florian Weimer 2018-01-31 10:35:44 UTC
(In reply to Petr Viktorin from comment #25)
> I understand the above correctly, you are suggesting that:
> - the $RPM_OPT_FLAGS should only be used for building Python, not
> extensions, and

$RPM_OPT_FLAGS and $RPM_LD_FLAGS (or any of the other mechanisms redhat-rpm-config provides) should be used when building with rpmbuild/mock, for Python itself and its extensions.  All distribution builds should use these flags as far as possible.

It would be best if the *current* value of these flags could be used.  This would allow us to use redhat-rpm-config to work around issues, without having to rebuild GCC or Python (for example, we can't rebuild python2 because of bug 1540316, and as a result, the workaround for gcc bug 1538648 is not active for Python extension modules).

> - We instruct setuptools to only use "-O2 -Wall -g -D_GNU_SOURCE -fPIC
> -fwrapv" for building extensions.
> 
> But what about things like -fexceptions -m64 -grecord-gcc-switches? Wouldn't
> it be good to use these for the extensions as well?

-m64 should not matter because GCC will use the right default.  -grecord-gcc-switches is used for marking builds, so most users will not care.  -fexceptions could be needed for compiling C extensions which are expected to interoperate with C++ exception handling.

I suppose we could maintain macros %{extension_cflags}, %{extension_cxxflags} and %{extension_ldflags} in redhat-rpm-config.  We then need to decide what to include there.  annobin would not be there.  I would perhaps drop -grecord-gcc-switches as well.  If we assume that extensions are always built in PIC mode and never linked statically, we will not need the external specs files.

But these would only be used when building outside of rpmbuild/mock.

> I understand it doesn't change the ABI. But I'm afraid users would lose
> warnings/optimizations/debug info. Is that just irrational fear?

For GCC itself, there is an expectation that users end up with upstream defaults if they just run gcc.  If Python users have different expectations, I think we can implement something else (but ideally, it would be consistent with other extension building tools).

Comment 27 Petr Viktorin (pviktori) 2018-01-31 10:49:40 UTC
No, Python extensions created with distutils are never linked statically (except when building or embedding Python itself).

Maintaining %{extension_*} in redhat-rpm-config would be easiest for us. However, I'm not sure how much this applies to other things than Python, and I'm worried that the issue is quite alien for people maintaining 
$RPM_OPT_FLAGS.
Do other extension building tools have this issue?

Another option would be to separate the flags from the Python spec – but to make that logic work properly it would need to fail if it sees new options it doesn't know about.

Comment 28 Florian Weimer 2018-01-31 11:38:49 UTC
(In reply to Petr Viktorin from comment #27)
> No, Python extensions created with distutils are never linked statically
> (except when building or embedding Python itself).

Thanks, that will simplify things.

> Maintaining %{extension_*} in redhat-rpm-config would be easiest for us.
> However, I'm not sure how much this applies to other things than Python, and
> I'm worried that the issue is quite alien for people maintaining 
> $RPM_OPT_FLAGS.

I don't think this an issue.

> Do other extension building tools have this issue?

Python 3 naturally has the same problem, and it seems that both Ruby and Perl are affected as well.  For Go, we'll probably have to do something similar.  Maybe npm as well.

Comment 29 Petr Viktorin (pviktori) 2018-01-31 12:09:22 UTC
Alright, if other languages are the same, then keeping it in redhat-rpm-config is the way to go.

So, to recap, we're just moving the specs and -grecord-gcc-switches?
That would mean:

%{extension_cflags}, i.e. flags for building outside rpmbuild/mock:
-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong --m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection

$RPM_OPT_FLAGS would be the above plus:
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -grecord-gcc-switches 

Python would then add (for both rpmbuild and third-party builds):
-D_GNU_SOURCE -fPIC -fwrapv


Once these are in redhat-rpm-config, I hope it'll be a simple change on the Python side.

I think you're the best person to do the redhat-rpm-config changes, but let me know if I can help!

Comment 30 Petr Viktorin (pviktori) 2018-06-21 15:07:08 UTC
From Bug #1543394:

> We realized that avoiding the dependency on annobin and redhat-rpm-config
> could not be implemented without changes to extension builders, and we
> received little support for that from the respective package maintainers.
> As a result, it does not make sense to add separate build flags in
> redhat-rpm-config.
> 
> Extension builders which hard-code the distribution CFLAGS should add a
> dependency on redhat-rpm-config and gcc (and gcc-c++ if applicable).

Comment 31 Miro Hrončok 2019-01-16 08:59:25 UTC
%{extension_cflags} now exist

see https://fedoraproject.org/wiki/Changes/Python_Extension_Flags

Comment 32 Miro Hrončok 2019-01-21 22:19:54 UTC

*** This bug has been marked as a duplicate of bug 1668029 ***


Note You need to log in before you can comment on or make changes to this bug.