Description of problem: ## Context Right now the redhat-rpm-config has %extension_* flags macros to be used by RPM spec files. I think the intent is to strip flag info defined in the redhat-rpm-config package. In case of rpms/ruby, the `rbconfig.rb` file is shipped in Ruby binary RPM, and it has compiled flags info, and it is used to build Ruby's native extension gem in the process of `gem install <gem name>`. Now we are trying to strip the flag info requiring redhat-rpm-config package from `rbconfig.rb`. This is related to https://bugzilla.redhat.com/show_bug.cgi?id=1284684 . In the process of `./configure` and `make, The `configure` script creates `config.status` file including flags info. The `make` creates `rbconfig.rb` file from `config.status` file. In ruby.spec, our first try was by the command [a] below. We expected the values of `%{build_cflags}`, `%{build_cxxflags}`, `%{build_ldflags}` are propagated to `config.status` then to `rbconfig.rb`. But it was a wrong assumption. Because in the configure script, the '-m64', '-m32' in CFLAGS, CXXFLAGS, LDFLAGS are removed. See https://github.com/ruby/ruby/blob/v3_1_1/tool/m4/ruby_universal_arch.m4#L7 . So, the first try was failed. [a] ``` sed -i \ -e 's|%{build_cflags}|%{extension_cflags}|g' \ -e 's|%{build_cxxflags}|%{extension_cxxflags}|g' \ -e 's|%{build_ldflags}|%{extension_ldflags}|g' \ %{buildroot}%{ruby_libarchdir}/rbconfig.rb ``` Another try is the command [b] below. [b] ``` sed -i \ -e '/CONFIG/ s|[ \t]\+-specs=[^ \t]\+| |g' \ -e '/CONFIG/ s|-Wl,[^ \t]*package_note[^ \t]*||g' \ %{buildroot}%{ruby_libarchdir}/rbconfig.rb ``` Ref. https://www.gnu.org/software/sed/manual/html_node/Regular-Expressions.html This command is referring the patterns used in %__extension_strip_flags. As I could not understand what `gsub(lto_flags_pattern, "")` is doing, I did not add the pattern to my sed command in the example above. > https://src.fedoraproject.org/rpms/redhat-rpm-config/blob/rawhide/f/macros#_106 ``` %__extension_strip_flags() %{lua: local name = rpm.expand("%{1}") local value = " " .. rpm.expand("%{build_" .. name .. "}") local specs_pattern = "%s+-specs=[^%s]+" local lto_flags_pattern = rpm.expand("%{?_lto_cflags}"):gsub("[%-%.]", "%%%1") local package_note_flags_pattern = "%-Wl,%S*package_note%S*" local result = value:gsub(specs_pattern, " "):gsub(lto_flags_pattern, ""):gsub(package_note_flags_pattern, "") print(result) } ``` And I added the following command [c] to check if the `rbconfig.rb` doesn't include redhat-rpm-config specific flags after the command above. [c] ``` grep %{_rpmconfigdir}/redhat %{buildroot}%{ruby_libarchdir}/rbconfig.rb && false ``` But the problem is the command [b] defined in ruby.spec is we don't want to manage the patterns to strip the flag info in rpms/ruby, because the `%__extension_strip_flags` can be changed in the future. I think the only redhat-rpm-config should manage the into to strip the flags info. ## Suggestion So, my ask is, shall we add the macro function to do command [b] (and [c]) in the `macros` file? The macro function could be Input argument 1: file path to be repaced. Input argument 2 (optional): pattern to be used to filter by `sed`. I am not familiar with the syntax of rpm macro and lua, here is an image by bash-like function. function strip_to_extension_flags_in_file() { local file=$1 local pattern=$2 sed -i \ -e "/${pattern}/ s|[ \t]\+-specs=[^ \t]\+| |g" \ -e "/${pattern}/ <lto_flags_pattern>" -e "/${pattern}/ s|-Wl,[^ \t]*package_note[^ \t]*||g" \ "${file}" } Version-Release number of selected component (if applicable): redhat-rpm-config-214-1.fc37.noarch How reproducible: Steps to Reproduce: 1. Here is the branch including command [a][b] https://src.fedoraproject.org/fork/jaruga/rpms/ruby/tree/wip/extension_flags . You can build the ruby.spec. Additional info:
Created attachment 1871374 [details] rbconfig.rb As a reference, I would attach the current original rbconfig.rb file that is before applying the `sed` command.
> As I could not understand what `gsub(lto_flags_pattern, "")` is doing, I did not add the pattern to my sed command in the example above. It removes -flto=auto -ffat-lto-objects
Several notes. 1) If the `%__extension_strip_flags()` was more generic accepting string and providing the modified output, that would be useful (and of course if it was not tainted by `__`). 2) If something like `{build_cflags} - %{extension_cflags}` was available, it would also make the filtering easier. 3) If the filter expressions or filtered values were available instead of being just implementation detail of `%__extension_strip_flags()`, that could also help.
I have trouble following what you are trying to do, sorry. I think this part is problematic: > The `configure` script creates `config.status` file including flags info. > The `make` creates `rbconfig.rb` file from `config.status` file. I think configure should use the official Fedora build flags (not the extension builder flags) for building Ruby and its extension modules. Furthermore, Ruby modules in other source RPMs should use the Fedora build flags as well. But if an end user builds a Ruby module, the extension builder flags should be used (perhaps unless the redhat-rpm-config package is installed). This is not just a nice-to-have wish list, the first two steps are required for Fedora build flags compliance. I don't quite see what more data you need from redhat-rpm-config to fix this on the Ruby side. I don't think it's a matter of flag string rewriting at this point, you need to add run-time flag discovery capabilities to the Ruby module build system. There really isn't any other way because fixed flags won't give you all three desirable outcomes (that is, correct Fedora Ruby build, correct Fedora Ruby modules build, relaxed flags for end user builds).
> I have trouble following what you are trying to do, sorry. What we are trying to do is what you said on the comment. > I think configure should use the official Fedora build flags (not the extension builder flags) for building Ruby and its extension modules. > > Furthermore, Ruby modules in other source RPMs should use the Fedora build flags as well. > > But if an end user build a Ruby module, the extension builder flags should be used (perhaps unless the redhat-rpm-config package is installed). * When we build ruby.spec to create rpms/ruby binary RPMs, and when we build rpms/rubygem-<something> to create the binary RPMs, we want to build it with Fedora build flags. * When end users run `gem install <a_gem_with_native_extension>` to build and install the gem, we want to build the gem with extension build flags without requiring the redhat-rpm-config RPM. And to achieve these requirements, first we want to modify the shipped rbconfig.rb (RbConfig) file in a ruby binary RPM, used when end users build and install a gem with native extension. Currently the file includes flags requiring redhat-rpm-config RPM package. ``` $ grep '/usr/lib/rpm/redhat' rbconfig.rb CONFIG["configure_args"] = " '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-rubylibprefix=/usr/share/ruby' '--with-archlibdir=/usr/lib64' '--with-rubyarchprefix=/usr/lib64/ruby' '--with-sitedir=/usr/local/share/ruby/site_ruby' '--with-sitearchdir=/usr/local/lib64/ruby/site_ruby' '--with-vendordir=/usr/share/ruby/vendor_ruby' '--with-vendorarchdir=/usr/lib64/ruby/vendor_ruby' '--with-rubyhdrdir=/usr/include' '--with-rubyarchhdrdir=/usr/include' '--with-sitearchhdrdir=$$(sitehdrdir)/$$(arch)' '--with-vendorarchhdrdir=$$(vendorhdrdir)/$$(arch)' '--with-rubygemsdir=/usr/share/rubygems' '--with-ruby-pc=ruby.pc' '--with-compress-debug-sections=no' '--disable-rpath' '--enable-shared' '--with-ruby-version=' '--enable-multiarch' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CC=gcc' 'CXX=g++' 'CFLAGS=-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' 'LDFLAGS=-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 ' 'CXXFLAGS=-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection'" CONFIG["DLDFLAGS"] = "-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 " CONFIG["CXXFLAGS"] = "-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection" CONFIG["LDFLAGS"] = "-L. -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -fstack-protector-strong -rdynamic -Wl,-export-dynamic -Wl,--no-as-needed" CONFIG["CFLAGS"] = "-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC" ``` Then when we build rpms/rubygem-<something> (e.g. rubygem-byebug) to create the binary RPMs, as the rubygem-<something>.spec use %gem_install macro to compile with the Fedora build flags, I expect that we can build the rubygem-* RPM with Fedora build flags. https://src.fedoraproject.org/rpms/rubygem-byebug/blob/rawhide/f/rubygem-byebug.spec#_87 %gem_install https://src.fedoraproject.org/rpms/ruby/blob/rawhide/f/macros.rubygems#_25 ``` %gem_install(d:n:) \ ... CONFIGURE_ARGS="--with-cflags='%{optflags}' --with-cxxflags='%{optflags}' --with-ldflags='%{build_ldflags}' $CONFIGURE_ARGS" \\\ gem install \\\ ... ```
> I think this part is problematic: > > > The `configure` script creates `config.status` file including flags info. > > The `make` creates `rbconfig.rb` file from `config.status` file. > > I think configure should use the official Fedora build flags (not the extension builder flags) for building Ruby and its extension modules. Maybe I understand what you think is a problem. Here are the steps of what we are trying to do. I think it's no problem. * We run the `configure` with Fedora flags not with extension flags in %build section. The /builddir/build/BUILD/ruby-3.1.1/config.status is created. * We run the `make` with Fedora flags, not with extension flags in %build section. The /builddir/build/BUILD/ruby-3.1.1/rbconfig.rb is created from the /builddir/build/BUILD/ruby-3.1.1/config.status . * We modify `%{buildroot}%{ruby_libarchdir}/rbconfig.rb` (/builddir/build/BUILDROOT/ruby-3.1.1-164.fc37.x86_64/usr/lib64/ruby/rbconfig.rb) in %install section. * We run the `make check` with Fedora flags, not with extension flags in %check section. The /builddir/build/BUILD/ruby-3.1.1/rbconfig.rb created from Fedora build flags is used.
I suggest you change rbocinfig.rb in the following ways: * Add Ruby code to check if /usr/lib/rpm/redhat/macros exists. If it does, set the variables by running “rpm --eval %build_flags” as appropriate *at run time* (not hard-coding the build flags). * If the check indicates that /usr/lib/rpm/redhat/macros does not exist, fall back to %extension_cflags etc., based on the values discovered at ruby package build time. /usr/lib/rpm/redhat/macros exists then, so these flags can be queried using a rpm command while generating rbconfig.rb. At least that's what I had in mind when I added the %extension_* macros. Trying to do this with string rewriting is going to be brittle, and won't be able to express the /usr/lib/rpm/redhat/macros file dependencies anyway.
(In reply to Florian Weimer from comment #7) First of all, the macros as currently designed assumes that if something is build with `%build_flags`, then the `%build_flags` string, as it was presented to configure script, appears in the build output. This assumption is wrong, because the build system is free to modify them. In Ruby case, the build system extracts the `-m64`/`-m32` just to store them elsewhere. > I suggest you change rbocinfig.rb in the following ways: > > * Add Ruby code to check if /usr/lib/rpm/redhat/macros exists. If it does, > set the variables by running “rpm --eval %build_flags” as appropriate *at > run time* (not hard-coding the build flags). > * If the check indicates that /usr/lib/rpm/redhat/macros does not exist, > fall back to %extension_cflags etc., based on the values discovered at ruby > package build time. /usr/lib/rpm/redhat/macros exists then, so these flags > can be queried using a rpm command while generating rbconfig.rb. Based on what I said earlier, there is no straight forward way to replace anything in rbconfig.rb by `%extension_cflags` > Trying to do this with string rewriting is going to be brittle, and won't be > able to express the /usr/lib/rpm/redhat/macros file dependencies anyway. The /usr/lib/rpm/redhat/macros on itself is not what is the issue. The problem are the compiler flags such as `-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1`. If we knew there is such option and we should treat something based on such option, then we could do this, but build system would need to tell us about it.
I don't think the current discussion is helpful. It is getting sidetracked. The RFE is to programmatically be able to see what flags/patterns are removed in %__extension_strip_flags. I think that is a reasonable request, however, I don't think we can just expose the Lua patterns. There is no way to use the Lua patterns via e.g. sed. I see several possible ways to achieve the end goal: 1) The code currently used in %__extension_strip_flags will generate both the Lua pattern and the sed expression (to be exposed in a macro) the implementation might be tricky, but both things will be consistent. 2) We add an alternate sed "implementation" of %__extension_strip_flags that we expose as a macro with the sed expression and write tests ensuring %__extension_strip_flags and the sed expression are consistent. 3) We would only keep the sed expression and we replace Lua pattern gsub with echo | sed. That might be slightly slower, but we would only have implementation.
> I suggest you change rbocinfig.rb in the following ways: > ... > At least that's what I had in mind when I added the %extension_* macros. As Vit said, the ways are difficult. I have no idea about how to do it. I can not find what parts in rbconfig.rb should be a placeholder to be replaced by the value of `rpm --eval %build_flags` or `rpm --eval %extension_cflags`.
For this topic, I got an idea to achieve our requirements from Ruby upstream. https://bugs.ruby-lang.org/issues/18691#note-1 A maintainer said that in Ruby configure and make, the values of CFLAGS (maybe CXXFLAGS, LDFLAGS too) are propagated to rbconfig.rb. But the value of ARCH_FLAG is only used in the building time of Ruby, and not propagated to rbconfig.rb. So, we might be able to run configure script like this in ruby.spec. I haven't checked it yet. ``` CFLAGS='%{extension_cflags}' CXXFLAGS='%{extension_cxxflags}' LDFLAGS='%{extension_ldflags}' \ ARCH_FLAG='%{build_cflags} %{build_cxxflags} %{build_ldflags}' \ %configure ``` > 2) If something like `{build_cflags} - %{extension_cflags}` was available, it would also make the filtering easier. But this is another reason why we want a macro describing `{build_cflags} - %{extension_cflags}` to set it to ARCH_FLAG.
This bug appears to have been reported against 'rawhide' during the Fedora Linux 37 development cycle. Changing version to 37.
This ticket is still for rawhide.
Here is my proposal to make this issue better, perhaps. It's not the best. I am looking for a better or simpler way. Construct build flags from extension flags and Red Hat RPM dependent flags. https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/222
This bug appears to have been reported against 'rawhide' during the Fedora Linux 38 development cycle. Changing version to 38.
This bug appears to have been reported against 'rawhide' during the Fedora Linux 39 development cycle. Changing version to 39.