Bug 2073170

Created attachment 1871374 [details]
rbconfig.rb

As a reference, I would attach the current original rbconfig.rb file that is before applying the `sed` command.

> As I could not understand what `gsub(lto_flags_pattern, "")` is doing, I did not add the pattern to my sed command in the example above.

It removes -flto=auto -ffat-lto-objects

Several notes.

1) If the `%__extension_strip_flags()` was more generic accepting string and providing the modified output, that would be useful (and of course if it was not tainted by `__`).
2) If something like `{build_cflags} - %{extension_cflags}` was available, it would also make the filtering easier.
3) If the filter expressions or filtered values were available instead of being just implementation detail of `%__extension_strip_flags()`, that could also help.

I have trouble following what you are trying to do, sorry.

I think this part is problematic:

> The `configure` script creates `config.status` file including flags info.
> The `make` creates `rbconfig.rb` file from `config.status` file.

I think configure should use the official Fedora build flags (not the extension builder flags) for building Ruby and its extension modules.

Furthermore, Ruby modules in other source RPMs should use the Fedora build flags as well.

But if an end user builds a Ruby module, the extension builder flags should be used (perhaps unless the redhat-rpm-config package is installed).

This is not just a nice-to-have wish list, the first two steps are required for Fedora build flags compliance. I don't quite see what more data you need from redhat-rpm-config to fix this on the Ruby side. I don't think it's a matter of flag string rewriting at this point, you need to add run-time flag discovery capabilities to the Ruby module build system. There really isn't any other way because fixed flags won't give you all three desirable outcomes (that is, correct Fedora Ruby build, correct Fedora Ruby modules build, relaxed flags for end user builds).

> I have trouble following what you are trying to do, sorry.

What we are trying to do is what you said on the comment.

> I think configure should use the official Fedora build flags (not the extension builder flags) for building Ruby and its extension modules.
>
> Furthermore, Ruby modules in other source RPMs should use the Fedora build flags as well.
>
> But if an end user build a Ruby module, the extension builder flags should be used (perhaps unless the redhat-rpm-config package is installed).

* When we build ruby.spec to create rpms/ruby binary RPMs, and when we build rpms/rubygem-<something> to create the binary RPMs, we want to build it with Fedora build flags.
* When end users run `gem install <a_gem_with_native_extension>` to build and install the gem, we want to build the gem with extension build flags without requiring the redhat-rpm-config RPM.

And to achieve these requirements, first we want to modify the shipped rbconfig.rb (RbConfig) file in a ruby binary RPM, used when end users build and install a gem with native extension. Currently the file includes flags requiring redhat-rpm-config RPM package.

```
$ grep '/usr/lib/rpm/redhat' rbconfig.rb
  CONFIG["configure_args"] = " '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-rubylibprefix=/usr/share/ruby' '--with-archlibdir=/usr/lib64' '--with-rubyarchprefix=/usr/lib64/ruby' '--with-sitedir=/usr/local/share/ruby/site_ruby' '--with-sitearchdir=/usr/local/lib64/ruby/site_ruby' '--with-vendordir=/usr/share/ruby/vendor_ruby' '--with-vendorarchdir=/usr/lib64/ruby/vendor_ruby' '--with-rubyhdrdir=/usr/include' '--with-rubyarchhdrdir=/usr/include' '--with-sitearchhdrdir=$$(sitehdrdir)/$$(arch)' '--with-vendorarchhdrdir=$$(vendorhdrdir)/$$(arch)' '--with-rubygemsdir=/usr/share/rubygems' '--with-ruby-pc=ruby.pc' '--with-compress-debug-sections=no' '--disable-rpath' '--enable-shared' '--with-ruby-version=' '--enable-multiarch' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CC=gcc' 'CXX=g++' 'CFLAGS=-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' 'LDFLAGS=-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 ' 'CXXFLAGS=-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection'"
  CONFIG["DLDFLAGS"] = "-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 "
  CONFIG["CXXFLAGS"] = "-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection"
  CONFIG["LDFLAGS"] = "-L. -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -fstack-protector-strong -rdynamic -Wl,-export-dynamic -Wl,--no-as-needed"
  CONFIG["CFLAGS"] = "-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC"
```

Then when we build rpms/rubygem-<something> (e.g. rubygem-byebug) to create the binary RPMs, as the rubygem-<something>.spec use %gem_install macro to compile with the Fedora build flags, I expect that we can build the rubygem-* RPM with Fedora build flags.

https://src.fedoraproject.org/rpms/rubygem-byebug/blob/rawhide/f/rubygem-byebug.spec#_87
%gem_install

https://src.fedoraproject.org/rpms/ruby/blob/rawhide/f/macros.rubygems#_25
```
%gem_install(d:n:) \
...	
CONFIGURE_ARGS="--with-cflags='%{optflags}' --with-cxxflags='%{optflags}' --with-ldflags='%{build_ldflags}' $CONFIGURE_ARGS" \\\
gem install \\\
...
```

> I think this part is problematic:
>
> > The `configure` script creates `config.status` file including flags info.
> > The `make` creates `rbconfig.rb` file from `config.status` file.
>
> I think configure should use the official Fedora build flags (not the extension builder flags) for building Ruby and its extension modules.

Maybe I understand what you think is a problem. Here are the steps of what we are trying to do. I think it's no problem.

* We run the `configure` with Fedora flags not with extension flags in %build section.
  The /builddir/build/BUILD/ruby-3.1.1/config.status is created.
* We run the `make` with Fedora flags, not with extension flags in %build section.
  The /builddir/build/BUILD/ruby-3.1.1/rbconfig.rb is created from the /builddir/build/BUILD/ruby-3.1.1/config.status .
* We modify `%{buildroot}%{ruby_libarchdir}/rbconfig.rb` (/builddir/build/BUILDROOT/ruby-3.1.1-164.fc37.x86_64/usr/lib64/ruby/rbconfig.rb) in %install section.
* We run the `make check` with Fedora flags, not with extension flags in %check section.
  The /builddir/build/BUILD/ruby-3.1.1/rbconfig.rb created from Fedora build flags is used.

I suggest you change rbocinfig.rb in the following ways:

* Add Ruby code to check if /usr/lib/rpm/redhat/macros exists. If it does, set the variables by running “rpm --eval %build_flags” as appropriate *at run time* (not hard-coding the build flags).
* If the check indicates that /usr/lib/rpm/redhat/macros does not exist, fall back to %extension_cflags etc., based on the values discovered at ruby package build time. /usr/lib/rpm/redhat/macros exists then, so these flags can be queried using a rpm command while generating rbconfig.rb.

At least that's what I had in mind when I added the %extension_* macros.

Trying to do this with string rewriting is going to be brittle, and won't be able to express the /usr/lib/rpm/redhat/macros file dependencies anyway.

(In reply to Florian Weimer from comment #7)

First of all, the macros as currently designed assumes that if something is build with `%build_flags`, then the `%build_flags` string, as it was presented to configure script, appears in the build output. This assumption is wrong, because the build system is free to modify them. In Ruby case, the build system extracts the `-m64`/`-m32` just to store them elsewhere.

> I suggest you change rbocinfig.rb in the following ways:
> 
> * Add Ruby code to check if /usr/lib/rpm/redhat/macros exists. If it does,
> set the variables by running “rpm --eval %build_flags” as appropriate *at
> run time* (not hard-coding the build flags).
> * If the check indicates that /usr/lib/rpm/redhat/macros does not exist,
> fall back to %extension_cflags etc., based on the values discovered at ruby
> package build time. /usr/lib/rpm/redhat/macros exists then, so these flags
> can be queried using a rpm command while generating rbconfig.rb.

Based on what I said earlier, there is no straight forward way to replace anything in rbconfig.rb by `%extension_cflags`

> Trying to do this with string rewriting is going to be brittle, and won't be
> able to express the /usr/lib/rpm/redhat/macros file dependencies anyway.

The /usr/lib/rpm/redhat/macros on itself is not what is the issue. The problem are the compiler flags such as `-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1`. If we knew there is such option and we should treat something based on such option, then we could do this, but build system would need to tell us about it.

I don't think the current discussion is helpful. It is getting sidetracked. The RFE is to programmatically be able to see what flags/patterns are removed in %__extension_strip_flags. I think that is a reasonable request, however, I don't think we can just expose the Lua patterns. There is no way to use the Lua patterns via e.g. sed. I see several possible ways to achieve the end goal:

 1) The code currently used in %__extension_strip_flags will generate both the Lua pattern and the sed expression (to be exposed in a macro) the implementation might be tricky, but both things will be consistent.
 2) We add an alternate sed "implementation" of %__extension_strip_flags that we expose as a macro with the sed expression and write tests ensuring %__extension_strip_flags and the sed expression are consistent.
 3) We would only keep the sed expression and we replace Lua pattern gsub with echo | sed. That might be slightly slower, but we would only have implementation.

> I suggest you change rbocinfig.rb in the following ways:
> ...
> At least that's what I had in mind when I added the %extension_* macros.

As Vit said, the ways are difficult. I have no idea about how to do it. I can not find what parts in rbconfig.rb should be a placeholder
to be replaced by the value of `rpm --eval %build_flags` or `rpm --eval %extension_cflags`.

For this topic, I got an idea to achieve our requirements from Ruby upstream.
https://bugs.ruby-lang.org/issues/18691#note-1

A maintainer said that in Ruby configure and make, the values of CFLAGS (maybe CXXFLAGS, LDFLAGS too) are propagated to rbconfig.rb. But the value of ARCH_FLAG is only used in the building time of Ruby, and not propagated to rbconfig.rb.

So, we might be able to run configure script like this in ruby.spec. I haven't checked it yet.

```
CFLAGS='%{extension_cflags}' CXXFLAGS='%{extension_cxxflags}' LDFLAGS='%{extension_ldflags}' \
  ARCH_FLAG='%{build_cflags} %{build_cxxflags} %{build_ldflags}' \
  %configure
```

> 2) If something like `{build_cflags} - %{extension_cflags}` was available, it would also make the filtering easier.

But this is another reason why we want a macro describing `{build_cflags} - %{extension_cflags}` to set it to ARCH_FLAG.

This bug appears to have been reported against 'rawhide' during the Fedora Linux 37 development cycle.
Changing version to 37.

This ticket is still for rawhide.

Here is my proposal to make this issue better, perhaps. It's not the best. I am looking for a better or simpler way.

Construct build flags from extension flags and Red Hat RPM dependent flags.
https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/222

This bug appears to have been reported against 'rawhide' during the Fedora Linux 38 development cycle.
Changing version to 38.

This bug appears to have been reported against 'rawhide' during the Fedora Linux 39 development cycle.
Changing version to 39.