RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1774139 - Please update redhat-rpm-config to include latest Forge macros
Summary: Please update redhat-rpm-config to include latest Forge macros
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: redhat-rpm-config
Version: 8.4
Hardware: Unspecified
OS: Unspecified
medium
unspecified
Target Milestone: rc
: 8.0
Assignee: Florian Festi
QA Contact: Radek Bíba
URL:
Whiteboard:
Depends On:
Blocks: 1759830 1945455
TreeView+ depends on / blocked
 
Reported: 2019-11-19 16:58 UTC by Robert-André Mauchin 🐧
Modified: 2021-04-01 00:50 UTC (History)
18 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-03-23 08:40:03 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
macros.forge diff betweeh rhel-8 and fedora-32 (14.96 KB, patch)
2020-02-18 11:34 UTC, Panu Matilainen
no flags Details | Diff

Description Robert-André Mauchin 🐧 2019-11-19 16:58:11 UTC
Hello,

In Fedora 31, we adopted new Golang macros https://fedoraproject.org/wiki/Changes/Adopt_new_Go_Packaging_Guidelines

These macros depends on the go-rpm-macros package https://src.fedoraproject.org/rpms/go-rpm-macros, which itself depends on the availability of the latest Forge macros from redhat-rpm-config (common.lua, forge.lua, macros.forge, macros.fedora-misc: i.e. this commit and all subsequent modifications to the aforementioned files: https://src.fedoraproject.org/rpms/redhat-rpm-config/c/c70110c677904a188d66ffd01a9ab024469ca62d )

Would it be possible to sync the RHEL package with Fedora to use these newer Forge macros, so we can package go-rpm-macros in EPEL?

The availability of go-rpm-macros would allow the GO SIG team to import all of our Go packages which have all been updated in F31. 


Thank you.

Comment 1 Joe Doss 2020-02-17 02:05:41 UTC
Any chance this can be looked into? Having go-rpm-macros would make packaging Go packages for EPEL much easier. 

Thanks!

Comment 2 Terry Bowling 2020-02-17 15:43:39 UTC
Can you help clarify the list of benefits vs. risk/impact?  For example, would this work against the fedora/rhel minimization efforts by requiring new dependencies on golang or other packages?  Any other risks to build root or infrastructure?  Other benefits/risks to end users?

Comment 3 Robert-André Mauchin 🐧 2020-02-17 17:38:56 UTC
(In reply to Terry Bowling from comment #2)
> Can you help clarify the list of benefits vs. risk/impact?  For example,
> would this work against the fedora/rhel minimization efforts by requiring
> new dependencies on golang or other packages?  Any other risks to build root
> or infrastructure?  Other benefits/risks to end users?

There would not be a new dependency on Golang directly. Golang would be only required by Go packages via go-rpm-macros.
I can't assess the whole risk part as I'm not familiar with all the change done in redhat-rpm-config that might impact the distribution outside of Go, you might need to ask the maintainer in Fedora.
The benefit would be to bring the Go ecosystem to end users and be able to package Go app like rclone, dnscrypt-proxy and so on, unbundled.
One alternative might be to use Fedora modules to build app for EPEL, but this might get difficult due to numerous cyclic dependencies.

Comment 4 Troy Dawson 2020-02-17 22:02:27 UTC
Looking at the Minimization side for RHEL8.

Currently in RHEL8:

go-srpms-macros (source)
  BuildDependencies: None
===
go-srpms-macros (noarch binary)
  Size: 18k
  Dependencies (Requires): None
  WhatRequires: redhat-rpm-config


Currently in Fedora Rawhide (what is proposed):
go-rpms-macros (source)
  BuildDependencies: None
===
go-srpm-macros (noarch binary)
  Size: 25k
  Dependencies (Requires): redhat-rpm-config
  WhatRequires: redhat-rpm-config, go-rpm-macros, golang

go-rpm-templates (noarch binary)
  Size: 18k
  Dependencies (Requires): go-rpm-macros
  WhatRequires: None

go-rpm-macros (arch specific (x86_64) binary)
  Size: 32k
  Dependencies (Requires): golang, go-filesystem, go-srpm-macros, golist
  WhatRequires: go-rpm-templates, go2rpm

go-filesystem (arch specific (x86_64) binary)
  Size: 7k
  Dependencies (Requires): None
  WhatRequires: go-rpm-macros, golang*


Summary:
I think this will have little impact in size and/or packages that need to be built for RHEL8.  I give it a +1.

For building, there are no extra packages that are needed.

For installing, the package that is already in RHEL8 (go-srpm-macros), will not pull in any more packages than it already does.
For the new packages (go-rpm-macros, go-filesystem, go-rpm-templates) they will not be pulling in any packages that are not already in RHEL8.

Comment 5 Panu Matilainen 2020-02-18 08:56:31 UTC
The risk/impact here is that major changes to those forge macros have the potential to break existing RHEL, EPEL and other 3rd party/customer packages.

It's a large pile of complicated Lua macros maintained by the community, so we don't have any real insight as to their backwards compatibility. It's entirely possible they are backwards compatible but we don't know and cannot assume that, we need to carefully review and verify the situation before any decisions.

Comment 6 Panu Matilainen 2020-02-18 11:34:35 UTC
Created attachment 1663736 [details]
macros.forge diff betweeh rhel-8 and fedora-32

Just to get an idea of what we're talking about here, here's a diff of macros.forge between rhel-8 and fedora-32. Whether there are other changes that would need to be included I don't know.

Comment 7 Nicolas Mailhot 2020-02-18 12:00:53 UTC
It is quite excessive to warn about the rate of "community" backwards incompatible changez when the changes you point at to direct result of rpm/redhat-rpm-config maintainers requesting deep changes during the initial merge (making the macros multi-instanciable to be able to define multiple sources) all all the changes went through months/years of FPC and redhat-rpm-config review

(some parts of the 2018! change are still stuck in PR review, mostly documentation-side)

Comment 8 Panu Matilainen 2020-02-18 12:11:26 UTC
Nicolas, I think you misunderstood. I'm not criticizing the changes, just stating a fact: there is a non-trivial amount of change involved, and that I personally have no clue about its compatibility. Assessing the compatibility/risk is something that simply must be done for *any* RHEL change.

Comment 9 Nicolas Mailhot 2020-02-18 12:29:06 UTC
There was a major change in late 2018 that was discussed and audited to death FCP and redhat-rpm-config side

https://src.fedoraproject.org/rpms/redhat-rpm-config/c/c70110c677904a188d66ffd01a9ab024469ca62d

It initially landed in FC30
https://src.fedoraproject.org/rpms/redhat-rpm-config/c/39fa88418c95a7943a900a0f68d9fa2691398375

That means that any Fedora spec that calls those macros, directly or indirectly (almost all Go packages, soonish most major Font packages now FPC greenlighted the guidelines changes) and was built for FC31 and later expects this version of the code and can not easily be backported to EL8/EPEL8 unless the redhat-rpm-config state here moves forward.

And it’s not so much because the changes were incompatible (they are 90% compatible except in weird corner cases that were cleaned up in the refactoring). It’s because they add new features and macro switches which are used by post FC30 specs.

specs that do not call the macros do not care, of course.

Changes since then are mostly minor bugfixes or feature enhancements. For example the pagure driver was added in

https://src.fedoraproject.org/rpms/redhat-rpm-config/c/61ec2ef5ba8ff9c8f8da70ad89565fd63797da94

It could not be done in 2018 because at that time, pagure was lacking the corresponding API.

Comment 10 Panu Matilainen 2020-02-18 12:36:30 UTC
> And it’s not so much because the changes were incompatible (they are 90% compatible except in weird corner cases that 
> were cleaned up in the refactoring). 

If you can point out the incompatibilities you know about, that'd be very useful information.

Comment 11 Nicolas Mailhot 2020-02-18 13:29:23 UTC
Honestly, after all this time I don’t remember. They were all very obscure corner cases that were never supposed to work but could be triggered with weird parameters combinations. I don’t think we had a single bug report in Fedora from someone that actually used those combinations (or maybe some people hit it but just streamlined their specs after getting pointed to the docs still stuck in FPC & redhat-rpm-config PRs)

There was one vocal complaint that the new macro version generated different source URLs for github, and the URLs were wrong anyway… It stopped after pointing out that the new version generated the exact URLs the person wanted, that it was the old version that was in the wrong, and the old version (the one in EL8 right now) was in the wrong because at the time it was written our packaging guidelines mandated this URL style (our guidelines changed before the rewrite).

So practically the only known effect is that if one constructed a spec with the old macro version (before FC31), with a github source, the new version will compute a different source URL, and koji will require uploading a new source archive to the lookaside cache (or changing the spec to force the old URL construct).

Anything QAed post-FC30 will have lookaside sources pointing to the new path.

Anyway, forge URLs point to github/gitlab/pagure download APIs. Changing those APIs (or pointing to a new one like FPC did for github) requires rewriting all the affected source paths, with or without macros.

Comment 12 Panu Matilainen 2020-02-19 08:55:47 UTC
Ack, thanks.

Comment 15 Nicolas Mailhot 2020-06-30 08:56:34 UTC
Hi Florian,

The current forge code works well and has not caused any problem for several Fedora releases.

Note that Fedora 33 (or 34, depending how things run out) will include the next major version of this code, that will add features and (hopefully) robustness, but will obviously not be initially as battle tested as the current code.

I hope to post the corresponding redhat-rpm-config (and possibly F33 change page) today

Comment 16 Florian Festi 2020-07-06 15:29:52 UTC
I really don't see us updating to the F33/34 version of the macros in the RHEL 8.4 time frame if those make it into Fedora only now.

Looking at the changes: From the outside there are only very few macros. The changes add and remove parameters, though. This is worrisome as there is the expectation that 3rd part packages continue to build. Parameter dropped include:

%forgemeta drops -p
%forgesetup(a:b:cDn:Tq) becomes %forgesetup(az:v) basically swapping out all params except -z creating a completely new command
%forgeautosetup(a:b:cDn:TvNS:p:) becomes %forgeautosetup(z:vNS:p:q) dropping -a -b -c -D -n -T

Basically both forgesetup and forgeautosetup stop being "drop-in" replacements for %setup/%autosetup. While such a change may be not a big deal for a distribution during a development cycle this is not the kind of change we are aiming for in an RHEL update release. Is adding back those parameters an option?

Comment 17 Nicolas Mailhot 2020-07-06 16:20:54 UTC
Yes, sure, I was not proposing to move EL8 to brand new Fedora code, just notting that the existing code, is now old enough its Fedora lifetime reaches its end.

As for your questions:

1. the -p was an artefact of primitive  early forge/go integration. It’s no longer used at all by the Go macros today and to my knowledge no one else ever used it. You can see in
https://src.fedoraproject.org/rpms/redhat-rpm-config/c/c314c8a28540c222d2629f4d1b54573c2c41715c?branch=master it is just a way to inhibit s. Normal users won’t pass s and inhibit with a p just after. 

You could reintroduce -p, in forgemeta, it’s quite useless but easy and safe to do.


2. the other dropped flags correspond to %setup flags already computed by forgemeta. Before this change it was possible to have a %setup flag passed twice, once by  %forgemeta built-in processing, and another time via manual CLI flags, with quite a lot of not so funny flag masking effects. The flags were removed to avoid those problems. You could always modify the %{forgesetupargs} variable and change its computed list of flags if you did not like it — that is a lot safer than stacking computed and manual flags and hoping the rpm arg parser won’t make a mess of things.

The removal of those flags from the macro interface was done after checking each flag to see it if would not conflict with auto-computed flags.

Generally speaking the code has been moving away from long lists of macro flags the rpm argument parser has difficulty processing, and towards global macros that are easier to manipulate for rpm, macro writers, and packagers.

You could reintroduce those flags, but the only case they do something, is when the user performs dangerous flag stacking, that may work by chance once but unlikely to work long term. And, their very existence was confusing packagers, and inducing them to try this kind of dangerous stacking.

Comment 18 Robert-André Mauchin 🐧 2020-07-06 18:07:11 UTC
(In reply to Florian Festi from comment #16)
> I really don't see us updating to the F33/34 version of the macros in the
> RHEL 8.4 time frame if those make it into Fedora only now.
> 
> Looking at the changes: From the outside there are only very few macros. The
> changes add and remove parameters, though. This is worrisome as there is the
> expectation that 3rd part packages continue to build. Parameter dropped
> include:
> 
> %forgemeta drops -p
> %forgesetup(a:b:cDn:Tq) becomes %forgesetup(az:v) basically swapping out all
> params except -z creating a completely new command
> %forgeautosetup(a:b:cDn:TvNS:p:) becomes %forgeautosetup(z:vNS:p:q) dropping
> -a -b -c -D -n -T
> 
> Basically both forgesetup and forgeautosetup stop being "drop-in"
> replacements for %setup/%autosetup. While such a change may be not a big
> deal for a distribution during a development cycle this is not the kind of
> change we are aiming for in an RHEL update release. Is adding back those
> parameters an option?


To be fair I don't see myself rebootstrapping the entire Go ecosystem for both F34 and EPEL8. I'll see if it's possible to build binary packages from Rawhide for EPEL8, like Rust does (with the side tag seeding thingie).

Comment 19 Florian Festi 2020-07-14 07:39:58 UTC
Is putting the new version of the macros into a new EPEL package an option? As I understand it it will be really difficult to make the new version compatible with packages using those old parameters. It's hard for me to judge how common this case really is. But for RHEL it's not really acceptable to break builds. And I have trouble seeing how this can be avoided when updating the macros.

Comment 20 Nicolas Mailhot 2020-07-20 16:26:57 UTC
Putting the new version of the macros in an EPEL package would not simplify things, quite the contrary.

Unless you renamed the macros they would clash with the version in redhat-rpm-config. And if you renamed the macros you would create an awful mess that would require users to identify which version they want to use, understand the differences, and make sure to make no naming mistake when porting specs from previous EL releases or from Fedora.

Right now the backwards compatibility you’re concerned about is removal of a few flags that never worked reliably. Packagers use the removed flags or not. If they use them, they will get a clean parser error, and know the spec need changing. If they do not, things will continue to work reliably.

Loads better than something that seems to work, till you get a failure, and realise EL has *two* versions of the same macros, one of which contains known problems, one fixed, with the fixed version using a naming no one knows about.

Comment 22 Filipe Brandenburger 2020-09-04 23:12:20 UTC
I'm also interested in building Go packages on EPEL 8.

Reading through this Bugzilla, it's unclear to me what the best way forward would be...

1) Backport enough of the new features of the %forge* macros onto redhat-rpm-config shipped with RHEL 8 / c8 / c8s, so that we could make go-rpm-macros 3.0.9 work there and use it to build Go packages?

2) Reimplement parts of go-rpm-macros that depend on the new features of %forge*, so that we would have a usable go-rpm-macros in EPEL 8 that could be used to build the other Go packages (potentially using latest Fedora specfiles)?

3) Something else that doesn't involve fixing the Go macros? I saw this mention of "build binary packages from Rawhide for EPEL8" but not sure how this would work or how to submit such packages into EPEL...

Anyways, I'm happy to help here. I'd appreciate some pointers of what would be worth exploring next. Happy to dig into that deep hole and try to come up with patches.

Cheers!
Filipe

Comment 24 Robert-André Mauchin 🐧 2020-12-20 01:22:36 UTC
> 3) Something else that doesn't involve fixing the Go macros? I saw this mention of "build binary packages from Rawhide for EPEL8" but not sure how this would work or how to submit such packages into EPEL...

Using something similar to Rust on Side Tags by Igor https://gist.github.com/eclipseo/63f52eb51535a4c90908f5f3a3caaf09

Comment 25 Florian Festi 2021-03-23 08:40:03 UTC
I really don't see us back porting the macros into the redhat-rpm-config package. Another option might be moving them into their own sub package. Then this sub package could be replaced by one with the newer version. Having such a package probably still violates the policy of some repositories. But may be there is a place where such a package (rpm-macros-newforge) could live. If this is something that would help the situation please reopen this bug (or a new one.)


Note You need to log in before you can comment on or make changes to this bug.