Bug 1634784

How do we know which CFLAGS are for Fedora builds only, and which ones should be used when building extensions? Currently we get them all in $RPM_OPT_FLAGS.

Once that is answered the implementation should be straightforward. There is a mechanism in Python to use separate sets of flags for these.

(In reply to Petr Viktorin from comment #1)
> How do we know which CFLAGS are for Fedora builds only, and which ones
> should be used when building extensions? Currently we get them all in
> $RPM_OPT_FLAGS.
> 
> Once that is answered the implementation should be straightforward. There is
> a mechanism in Python to use separate sets of flags for these.

The problem is that extensions in Fedora must use the Fedora build flags.  So you need to add some custom logic to the extension builder to detect whether it is running under rpmbuild (maybe due to the presence of special packages or files on the system) and alter behavior.

I think it was the Ruby who did not want to do that under any circumstances, which pretty much killed this approach, and we are now adding a dependency on redhat-rpm-config and annobin to all extension builders.

(In reply to Florian Weimer from comment #2)
> (In reply to Petr Viktorin from comment #1)
> > How do we know which CFLAGS are for Fedora builds only, and which ones
> > should be used when building extensions? Currently we get them all in
> > $RPM_OPT_FLAGS.
> > 
> > Once that is answered the implementation should be straightforward. There is
> > a mechanism in Python to use separate sets of flags for these.
> 
> The problem is that extensions in Fedora must use the Fedora build flags. 
> So you need to add some custom logic to the extension builder to detect
> whether it is running under rpmbuild (maybe due to the presence of special
> packages or files on the system) and alter behavior.

Is setting extra CFLAGS (and other env vars) not enough?

> I think it was the Ruby who did not want to do that under any circumstances,
> which pretty much killed this approach, and we are now adding a dependency
> on redhat-rpm-config and annobin to all extension builders.

I don't see how that means we can't have two separate sets of flags. If the union of the two sets is put in $RPM_OPT_FLAGS, Ruby can continue to use that.
But I don't want to maintain Python-specific code to separate the two sets out of $RPM_OPT_FLAGS.

(In reply to Petr Viktorin from comment #3)
> (In reply to Florian Weimer from comment #2)
> > (In reply to Petr Viktorin from comment #1)
> > > How do we know which CFLAGS are for Fedora builds only, and which ones
> > > should be used when building extensions? Currently we get them all in
> > > $RPM_OPT_FLAGS.
> > > 
> > > Once that is answered the implementation should be straightforward. There is
> > > a mechanism in Python to use separate sets of flags for these.
> > 
> > The problem is that extensions in Fedora must use the Fedora build flags. 
> > So you need to add some custom logic to the extension builder to detect
> > whether it is running under rpmbuild (maybe due to the presence of special
> > packages or files on the system) and alter behavior.
> 
> Is setting extra CFLAGS (and other env vars) not enough?

It requires writing logic in the extension builder to evaluate them.  Usually, they are not sensitive to CFLAGS at all.  Is Python different?

> > I think it was the Ruby who did not want to do that under any circumstances,
> > which pretty much killed this approach, and we are now adding a dependency
> > on redhat-rpm-config and annobin to all extension builders.
> 
> I don't see how that means we can't have two separate sets of flags. If the
> union of the two sets is put in $RPM_OPT_FLAGS, Ruby can continue to use
> that.
> But I don't want to maintain Python-specific code to separate the two sets
> out of $RPM_OPT_FLAGS.

That's reasonable.  I still may have patches somewhere which separate the flags, specifically around annobin usage.

Bug 1543394 comment 7 shows a potential patch for redhat-rpm-config.

Adrian, I think this also covers your other requirement, compiling Python extensions with non-GCC compilers which do not understand -specs=.  Right?

(In reply to Florian Weimer from comment #4)
> (In reply to Petr Viktorin from comment #3)
> > (In reply to Florian Weimer from comment #2)
> > > (In reply to Petr Viktorin from comment #1)
> > > > How do we know which CFLAGS are for Fedora builds only, and which ones
> > > > should be used when building extensions? Currently we get them all in
> > > > $RPM_OPT_FLAGS.
> > > > 
> > > > Once that is answered the implementation should be straightforward. There is
> > > > a mechanism in Python to use separate sets of flags for these.
> > > 
> > > The problem is that extensions in Fedora must use the Fedora build flags. 
> > > So you need to add some custom logic to the extension builder to detect
> > > whether it is running under rpmbuild (maybe due to the presence of special
> > > packages or files on the system) and alter behavior.
> > 
> > Is setting extra CFLAGS (and other env vars) not enough?
> 
> It requires writing logic in the extension builder to evaluate them. 
> Usually, they are not sensitive to CFLAGS at all.  Is Python different?

It should be. We should fix it if it's not.

Florian, how can we help move this forward?

(In reply to Petr Viktorin from comment #7)
> Florian, how can we help move this forward?

See comment 5.  I'm waiting for comments from Adrian.

I had a look at that patch, but I think I do not understand it. This seems to be for building with RPM. Which definitely fix my problem if it no longer sets '-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1' .

As I reported that '-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1' is set from one of those files:

/usr/lib64/python3.6/_sysconfigdata_dm_linux_x86_64-linux-gnu.py
/usr/lib64/python3.6/_sysconfigdata_m_linux_x86_64-linux-gnu.py
/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu/Makefile
/usr/libexec/platform-python3.6m-x86_64-config

It is not clear how changing RPM flags would help in my case.

(In reply to Adrian Reber from comment #9)
> I had a look at that patch, but I think I do not understand it. This seems
> to be for building with RPM. Which definitely fix my problem if it no longer
> sets '-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1' .
> 
> As I reported that '-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1' is set
> from one of those files:
> 
> /usr/lib64/python3.6/_sysconfigdata_dm_linux_x86_64-linux-gnu.py
> /usr/lib64/python3.6/_sysconfigdata_m_linux_x86_64-linux-gnu.py
> /usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu/Makefile
> /usr/libexec/platform-python3.6m-x86_64-config
> 
> It is not clear how changing RPM flags would help in my case.

My intent was that roughly the following would happen (using Python here as an example):

* Python would stop hard-coding the current flags from redhat-rpm-config.

* Instead, Python would store flags from redhat-rpm-config designed for building extensions with various toolchains, providing reduced hardening coverage and Fedora build flags compliance.  These would be used for building extensions using the usual Python mechanisms.

* But for distribution RPM builds, Python would still use the standard redhat-rpm-config build flags from the *current* build environment (and not the flags captured at Python build time).  This ensures hardening coverage and build flags compliance for the Python extension modules we build ourselves.

* If an end user wishes to follow our policies, they can use a mock buildroot or trigger whatever mechanism is used to switch to the distribution build behavior.

I hope this clarifies why a second set of flags is desirable in this context.

Florian, this all sounds like it would work for at least what I am looking for. Thanks for finding a solution.

Is there any way for me to help move this forward?

Florian Weimer updated his redhat-rpm-config patch: https://bugzilla.redhat.com/show_bug.cgi?id=1543394#c13

Experimental PR for Fedora: https://src.fedoraproject.org/rpms/python3/pull-request/75#

This is now in Rawhide (Rawhide's redhat-rpm-config >= 127; note that versioning in other Fedoras may be different).

It can be ported to RHEL8.

FWIW, as an expected side effect it seems this change may cause python modules packaged in RHEL to lose annobin coverage after rebuild if they don't add the rpm CFLAGS to the flags returned by python3-config --cflags.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:3520