Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2216918

Summary: rhel 9 gcc-toolset-12-annobin newer than centos stream 9
Product: Red Hat Enterprise Linux 9 Reporter: Germano Massullo (Thetra) <germano.massullo>
Component: gcc-toolset-12-annobinAssignee: Nick Clifton <nickc>
Status: CLOSED CURRENTRELEASE QA Contact: Václav Kadlčík <vkadlcik>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: CentOS StreamCC: bstinson, carl, fweimer, jwboyer, markobri, mcermak, nickc, vkadlcik
Target Milestone: rcKeywords: Bugfix, Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: gcc-toolset-12-annobin-11.08-5.el9_2 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-09-28 06:22:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Germano Massullo (Thetra) 2023-06-23 08:31:07 UTC
this el9-next build of darktable is failing
https://kojipkgs.fedoraproject.org//work/tasks/8693/102448693/build.log
https://koji.fedoraproject.org/koji/taskinfo?taskID=102448693

when instead this el9 is successful
https://kojipkgs.fedoraproject.org//packages/darktable/4.4.0/1.el9/data/logs/aarch64/build.log
https://koji.fedoraproject.org/koji/buildinfo?buildID=2218776

a senior dev in #epel channel said that the issue may be a discrepancy between rhel9 and c9 for gcc-toolset-12-annobin-plugin-gcc.  rhel9 has 11.08-2.el9, but that build failed for c9 and epel9-next is still building against 10.76-2.el9 which appears to be broken.
Later he also tried the darktable build against internal rhel 9.3 + epel + epel-next, and it worked. So it's fairly sure the problem is that out-of-sync gcc-toolset-12-annobin package.

Comment 1 Nick Clifton 2023-06-23 14:02:34 UTC
(In reply to Germano Massullo from comment #0)
 
> a senior dev in #epel channel said that the issue may be a discrepancy
> between rhel9 and c9 for gcc-toolset-12-annobin-plugin-gcc.  rhel9 has
> 11.08-2.el9, but that build failed for c9 and epel9-next is still building
> against 10.76-2.el9 which appears to be broken.

This does indeed appear to be the case.

How is annobin for el9-next built ?  Is there a way to trigger a rebuild ?

Comment 2 Carl George 🤠 2023-06-28 21:41:23 UTC
epel9-next is a layered repo on top of epel9 to ensure compatibility with CentOS Stream 9.  While epel9 builds against RHEL 9, epel9-next builds against CentOS Stream 9.  I believe the problem here is that RHEL 9 has gcc-toolset-12-annobin-plugin-gcc-11.08-2.el9, but CentOS Stream still has gcc-toolset-12-annobin-plugin-gcc-10.76-2.el9.  It seems that the root of the issue is this failed build:

https://kojihub.stream.centos.org/koji/buildinfo?buildID=33575

Get that resolved and published in CentOS Stream 9, and it will sync to the private mirror Koji uses.  Then the epel9-next build of darktable should work.

Comment 4 Nick Clifton 2023-06-29 10:46:16 UTC
(In reply to Carl George 🤠 from comment #2)
> epel9-next is a layered repo on top of epel9 to ensure compatibility with
> CentOS Stream 9.  While epel9 builds against RHEL 9, epel9-next builds
> against CentOS Stream 9. 

Ah - thanks for the explanation.

> I believe the problem here is that RHEL 9 has
> gcc-toolset-12-annobin-plugin-gcc-11.08-2.el9, but CentOS Stream still has
> gcc-toolset-12-annobin-plugin-gcc-10.76-2.el9.  It seems that the root of
> the issue is this failed build:
> 
> https://kojihub.stream.centos.org/koji/buildinfo?buildID=33575

Indeed - CentOS has upgraded to Clang 16, whereas RHEL-9 is using Clang 15.
Clang 16 changed its API for certain features used by the annobin plugins for
Clang and LLVM, so the build no longer works.

> Get that resolved and published in CentOS Stream 9, and it will sync to the
> private mirror Koji uses.  Then the epel9-next build of darktable should
> work.

I have completed a build, but it still has to get through gating:

https://kojihub.stream.rdu2.redhat.com/koji/taskinfo?taskID=2424570

Comment 11 Nick Clifton 2023-07-21 07:52:45 UTC
OK, I have built gcc-toolset-12-annobin-11.08-4.el9_2 and it is currently in gating here:

  https://dashboard.osci.redhat.com/#/artifact/brew-build/aid/54098407

Comment 38 Nick Clifton 2023-09-05 10:14:01 UTC
Pushing out to RHEL 9.4 as this issue seems to be taking a long time to resolve.

Comment 39 Nick Clifton 2023-09-14 13:18:30 UTC
It looks like this is fixed now:

  $ rhel-repoquery --quiet --centos-stream 9 gcc-toolset-12-annobin-plugin-gcc
  AppStream-Centos-9-Stream-prod: gcc-toolset-12-annobin-plugin-gcc-11.08-5.el9.x86_64

  $ rhel-repoquery --quiet 9 gcc-toolset-12-annobin-plugin-gcc
  AppStream-9: gcc-toolset-12-annobin-plugin-gcc-11.08-2.el9.x86_64

Well almost.  It seems that the RHEL version is a couple of revisions behind (11.08-2 vs 11.08-5).
There is a build of the 11.08-5 revision for RHEL:

  https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=2618544

And the build passed through gating:

 https://dashboard.osci.redhat.com/#/artifact/brew-build/aid/54290008

So I am not sure why it has not reached the compose.

Moving to MODIFIED for now, in the hopes that this is just a temporary hiccup.

Comment 42 Václav Kadlčík 2023-09-19 10:08:23 UTC
This particular discrepancy between centos and rhel was eliminated
but it just made another problem visible. While gcc-toolset-12-annobin
is on par, gcc-toolset-12 in centos is behind rhel:

    centos: gcc-toolset-12-runtime-12.0-5.el9.x86_64
    rhel:   gcc-toolset-12-runtime-12.0-6.el9.x86_64

Unfortunately the centos version lacks the trigger mechanism to select
a suitable annobin. As a consequence, gcc-toolset-12 on centos is
"broken" from the rpmbuild point of view:

    /opt/rh/gcc-toolset-12/root/usr/bin/g++   -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection  -fPIE -o CMakeFiles/cmTC_c23d3.dir/testCXXCompiler.cxx.o -c /root/rpmbuild/BUILD/darktable-4.4.0/redhat-linux-build/CMakeFiles/CMakeTmp/testCXXCompiler.cxx
    cc1plus: fatal error: inaccessible plugin file /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/plugin/gcc-annobin.so expanded from short plugin name gcc-annobin: No such file or directory
    compilation terminated.

At this moment I'd propose:
  1.  Closing this ticket as CURRENTRELEASE
  2A. Opening another one to align gcc-toolset-12-runtime as well
      -- or --
  2B. Suggesting the reporter to switch the build to gcc-toolset-13
      (which is in much better shape)

Comment 43 Václav Kadlčík 2023-09-28 06:22:05 UTC
OK, per the previous comment I'm closing this as there's nothing what can be done in gcc-toolset-12-annobin. I've filed a follow-up ticket on the misaligned gcc-toolset-12-runtime: https://issues.redhat.com/browse/RHEL-10172