Note that maybe you want to give a try to SCM rpkg method, which is able to pregenerate certain parts of spec file during srpm generation phase including BuildRequires.

I am not sure how exactly BuildRequires for Go packages are generated but it would be very cool to include the macros for doing that in rpkg standard macro library.

I don't really mind a different method here and but maybe it would be good to join efforts here.

See https://docs.pagure.org/rpkg-util/spec_templates_from_scratch.html how rpkg spec templates work. But you will see there only the basic Git versioning macros.
I would be happy if we could make the library wider.

Note that I am planning to get support for this feature (spec templates) into Koji too (although you might be faster) so it will not stay limited to Copr Build System only.

So I am just mentioning this before we can take a proper look at this issue.

Hi,

At the point of the automation work you just have a 

%goprep

call in %prep that does the heavy lifting (source unpacking, setup of the GOPATH tree, call to the code analysis engine, relaying of the result to mock)

Though of course real-world upstream code is imperfect, in some cases full of garbage, so you may need some patching and removing before the result can be fed to code analysis and produce clean buildrequires.

I'm still working on defining the simpler and most effective cleanup patterns of upstream code. The basic tech works, I know how to fix code manually, I just need to formalize the usual way you clean up upstream mistakes.

So I believe there won't be any problem to include the result in rpkg. Just let me finish the human packaging template examples first :)

Ok, sure, just note that rpkg computes the resulting spec file at srpm build phase (and not rpm build phase) and it doesn't need mock or anything because it just statically renders computed values of BuildRequires into the resulting spec file e.g. like this:

BuildRequires:x
BuildRequires:y

The resulting rpm spec is then packed into srpm (together with sources) and only after that, the srpm with already computed BR values is passed to mock for a build.

So in case, we would like to include this into rpkg, we would need something that does all the code analysis and other needed logic and produces block of BuildRequires in the end:

BuildRequires: <pkg1>
BuildRequires: <pkg2>

to stdout.

There is no need to communicate with mock or anything. So actually packaging this will be slightly different. The rpkg macro will be e.g. {{{ goprep }}} and will be placed where BuildRequires are normally placed in a spec file.

I am not saying you should jump to do something else than you are currently doing but I think it might help to understand how rpkg actually may support this case with your help.

You can see in line 2500 of
https://paste.fedoraproject.org/paste/hglkYSWJAHd2YuGBX2XnsA

That deps are put in an env var before being passed to mock.

It needs to be in prep because you have all the unpack logic before, and that logic is tricky if you have a package that includes multiple sources, some of which are patched or fixed in some way in prep. Doing it in rpkg would require creating a simili-prep rpkg section in spec files.

Thus, I would be more interested if rpkg allowed me to do BuildRequires-like declarations in prep, and rpmbuild just picked up those at the end of prep, and it was transparent to the spec writer. %prep is more complex than the header section where BuildRequires are manually declarated right now, so simplifying this header section, at the cost of %prep complexity, is a huge no-no.

Also, I hope you realise the prepy things the macro does have their own requirements, so it's not just "rpkg will pregenerate at rpmbuild stage" it's also "rpkg will require all the stuff needed by prep to be in the default build root", unless you add a separate Requires level just for rpkg. Right now we have a nice system where one only needs a single macro-only package in the buildroot, and it adds transparently prep requirements for packages that need them only with the BR stage :

https://copr-be.cloud.fedoraproject.org/results/nim/go-macros-3/fedora-rawhide-x86_64/00812443-golang-github-shopspring-decimal/builder-live.log

→ nothing Go specific in the default buildroot except go-srpm-macros, that triggers a BR on go-rpm-macros for Go packages, and go-rpm-macro pulls in all the Go env (including the one used in %prep)

https://copr-be.cloud.fedoraproject.org/results/nim/fontpackages-2/fedora-rawhide-x86_64/00811647-sil-mondulkiri-extra-fonts/build.log.gz

→ nothing fonts specific in the default buildroot except fonts-srpm-macros, that triggers a BR on fonts-rpm-macros for fonts packages, and fonts-rpm-macro pulls in all the fonts env (including the one used in %prep)

(both macro sets use the same zero-default-buildroot-footprint except for macro files approcah)

So, I'm not saying it can't be done by rpkg, the concepts are close enough it would probably work with little adaptation, but there is clearly quite a lot of engineering to do to move rpkg from "interesting concept" to "ready for everyday mass use" stage. And I'm far from convinced it would end in a net simplification of fedora specs and infra.

Besides:

1. at this point builds work in plain rpmbuild and mock with pm_request, so we can do all the private builds we want, including for el7

2. the only question is how to push the result to Fedora, and that requires only activating a mock option in copr and koji, and mock is one of Fedora's standard infra build tools (even though this is probably slightly more work because the nswpawn guys block the socket pm_request uses, but that's just fixing nspawn and mock which is a win-win for Fedora)

3. the current syntax is familiar to every packager and reviewer, it's just a fooprep call in prep. There's no separate *setup call anymore

4. I have (and I'm not the only Go packager concerned in Fedora) 771 existing specs that need this

So, I'm not going to rush to rewrite 771 checked and QA spec files to use rpkg, and make them rpkg-specific, when rpkg is not part of Fedora standard build tools (so that would force every single Go packager in the distro to install, deploy and configure rpkg on their systems), the resulting spec syntax is non standard (so that would require rewriting many Fedora packaging guidelines, and forming reviewers on the new rpkg guidelines), and so on. Lots of work to rewrite specs, lots of work to QA the result, high chance rpkg is so alien to Fedora packaging customs the result will be rejected. Also, high chance the result will be more verbose specs, since the current solution is a one-line macro call in prep, and my primary objective is to make Go spec terser and easier to maintain for Fedora packagers.

So, the cheapest and easiest way for Fedora and me right now is to just use pm_request. That the Java SIG could probably use too as they wrote the thing in the first place.

And then in second stage you can arrive and show rpkg can make all those specs better, and get Fedora buy-in.

If you want to play with it I can sent you my Go spec stash as it is now. I'd have pointed you to a copr, which is how such engineering tests are supposed to be done, but I won't load copr with 700+ builds that I know will fail in the absence of pm-request.

The Go macros themselves are in
https://copr.fedorainfracloud.org/coprs/nim/go-macros-3/

You can ignore the golang-* packages, I stopped the build as soon as I saw pm request was absent so they're not complete.

It all works because @rh people sneaked bundled code in go-compilers, so the golist command the macros need is available, even though its built is completely non conformant with Fedora best practices (that's the king of thing the new macros are supposed to help eradicating).

Well, I think most of your points about adoption and usability of rpkg can be answered.

As for this RFE, pm-request enable/disable checkbox (similar to network-enabled) might be desired according to https://github.com/rpm-software-management/mock/wiki/Plugin-PMRequest docs. 

We should wait for https://github.com/rpm-software-management/mock/issues/218 to get resolved first though.

Sure, I don't think they're blocking from a technical POW, they're just a huge pile of work to do before rpkg can get viable as solution.

As for https://github.com/rpm-software-management/mock/issues/218 that's probably just a bad security rule in selinux or systemd, but no one systemd or selinux side is going to look at it if infra does not ask it. So if that's the only blocker please ask dwalsh or Lennart to look at it, I can do it too, but things will go faster if infra asks.

Longer term I've described a cleaner more integrated solution in
https://github.com/rpm-software-management/rpm/issues/104
and
https://github.com/rpm-software-management/mock/issues/160

but it needs changes at least in mock, and also in rpm if you want rpm querying to support the result (as some people do). Short to mid term pm request is the only existing deployable working solution.

It's totally orthogonal to rpkg use or not, I sure hope rpkg will be able to template-out the result, it just does not need rpkg to work at the most basic level.

(In reply to Nicolas Mailhot from comment #6)
> As for https://github.com/rpm-software-management/mock/issues/218 that's
> probably just a bad security rule in selinux or systemd, but no one systemd
> or selinux side is going to look at it if infra does not ask it. So if
> that's the only blocker please ask dwalsh or Lennart to look at it, I can do
> it too, but things will go faster if infra asks.

Could you help with debugging that issue?  Does that mean that you tried
to switch SELinux to Permissive and things were working fine?

> It's totally orthogonal to rpkg use or not, I sure hope rpkg will be able to
> template-out the result, it just does not need rpkg to work at the most
> basic level.

Agreed, dynamic BuildRequires have different scope than tools that generate
sources for SRPM (tito, rpkg, ..).

I'd be happy to run any test you want, to be honest I'm lost in all the layers of security here, so I don't quite know why to test.

I see mock creating the pm_request socket with or without nspawn

When nspawn is enabled, the pm request client can not talk to this socket anymore.

What tests do you want me to run?

Just permissive mode on the mock host? Something else that changes selinux settings in the buildroot?

Dunno, capabilities (of the process), permissions, ownership, ...?

I'd say that you should be able to start interactive /bin/bash process (from
%prep phase for example) and experiment with the socket.

I will retry but IIRC things running from spec the do not see at all the socket in nswpan mode. It's hidden by the security settings (selinux or container side). And one can see from the outside of the container, that mock did create the pm request socket in its usual place.

Closing in favor of Dynamic Build Dependencies, which just landed in Mock - and
should be soon in RPM.