Bug 1391889 - [rfe] Make the dist-git feature universal
Summary: [rfe] Make the dist-git feature universal
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Copr
Classification: Community
Component: backend
Version: unspecified
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: clime
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-11-04 10:33 UTC by Pavel Raiskup
Modified: 2020-05-05 09:20 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-11-02 11:52:54 UTC


Attachments (Terms of Use)

Description Pavel Raiskup 2016-11-04 10:33:30 UTC
I promised to Michal I'll submit an official ticket for this:

There's been done great progress on dist-git support in:
https://github.com/fedora-copr/copr/commit/308daed9235cb08da31fac4f5b3c7bbc932d1b4c

However there are some things that need to be done better.

Basically, "dist-git = git + lookaside cache", but the code requires only
git directory (not info about lookaside cache).

I would suggest having those "features", with this configuration:

  * DistGit
      git_url + lookaside_cache identificator + git_hash is needed to perform
      successful build
  * FedoraDistGit
      only component name (aka 'fedpkg clone PKG') + git_hash is neededj
  * RHELDistGit
      by default disabled (enabled in internal copr), only component name +
      git_hash is needed

I could implement the latest point, once the first two are available, but
the actual dist-git implementation has some things hardwired (e.g. fedpkg)
and making this work against internal dist git would be problem.  Also,
for internal purposes it would be very fine to have both RHEL and Fedora
dist-git support (not RHEL or Fedora only).

Ideally, Fedora/RHEL dist-gits could be just configurable options, so we could
add Mageia/CentOS dist-git easily too.

Comment 1 clime 2016-11-10 08:21:53 UTC
The first two points are now roughly covered by https://pagure.io/copr/copr/c/2174d0c7e9f3e0d2a99df6a203594146b4112323?branch=master. I did not include the git_hash yet but it can be definitely done. 

For the RHELDistGit the question is if we can also use fedpkg. It would be great if the answer was yes.

Comment 2 Pavel Raiskup 2016-11-10 10:34:47 UTC
(In reply to clime from comment #1)
> The first two points are now roughly covered by
> https://pagure.io/copr/copr/c/
> 2174d0c7e9f3e0d2a99df6a203594146b4112323?branch=master.

Thanks!  A bit detail is that this is still not very "generic" dist git,
that's still Fedora-like dist-git, but it is much closer to what we
necessarily need.  It would be much more general to avoid using 'fedpkg'
at all, but use pyrpkg directly.  Would that make sense?

> I did not include the git_hash yet but it can be definitely done.

Isn't that equivalent to 'branch'?  Maybe rename branch to hash, or some
more generic naming?

> For the RHELDistGit the question is if we can also use fedpkg. It would be
> great if the answer was yes.

That's now probably possible, but it is not easy to try quickly on my box,
have you tried to install 'rhpkg'?  There's now one fedpkg.conf jinja
template which has a lot of stuff hardcoded, which might cause problems.

One significant issue I see there is that we don't separate lookaside
cache and git clone url properly => Can some dist git instance have those
on different domains?

Can one dist-git instance have lookaside cache locate on different URI
than other dist-git instance (now it is /repo/pkgs)?

Also, for package testing, it would be very neat to have a possibility, for
example, to build spec file from github repository against Fedora's
lookaside cache.

Comment 3 Pavel Raiskup 2016-11-10 10:46:42 UTC
(In reply to Pavel Raiskup from comment #2)
> It would be much more general to avoid using 'fedpkg' at all, but use
> pyrpkg directly.  Would that make sense?

Note that once you hard-code fedpkg script into copr, you at least don't
have to bother your users with specifying anything else than 'component'
name and 'branch/git-hash'.

It would be supper nice to have a way to do:

 $ copr-cli build COPR --dist-git[=fedora] tar 28cb7b49

Or:

 $ copr-cli build COPR --dist-git=rhel tar rhel-7.4

Or, if those are not sufficient, fallback to:

 $ copr-cli build COPR \
      --dist-git-anon-clone-url https://github.com/user/project \
      --dist-git-lookaside      http://example.com/something \
      [--dist-git-hash md5]

Every other configuration for pyrpkg shouldn't be needed.

Comment 4 Pavel Raiskup 2016-11-10 12:33:57 UTC
(In reply to Pavel Raiskup from comment #3)
>  $ copr-cli build COPR --dist-git=rhel tar rhel-7.4

Of course Fedors's frontend (not client) should detect that
--dist-git=XXXX is valid/invalid, i.e. the logic implemented (configured)
on frontend?

Comment 5 clime 2017-11-02 11:52:54 UTC
(In reply to Pavel Raiskup from comment #0)
> I promised to Michal I'll submit an official ticket for this:
> 
> There's been done great progress on dist-git support in:
> https://github.com/fedora-copr/copr/commit/
> 308daed9235cb08da31fac4f5b3c7bbc932d1b4c
> 
> However there are some things that need to be done better.
> 
> Basically, "dist-git = git + lookaside cache", but the code requires only
> git directory (not info about lookaside cache).
> 
> I would suggest having those "features", with this configuration:
> 
>   * DistGit
>       git_url + lookaside_cache identificator + git_hash is needed to perform
>       successful build
>   * FedoraDistGit
>       only component name (aka 'fedpkg clone PKG') + git_hash is neededj
>   * RHELDistGit
>       by default disabled (enabled in internal copr), only component name +
>       git_hash is needed
> 
> I could implement the latest point, once the first two are available, but
> the actual dist-git implementation has some things hardwired (e.g. fedpkg)
> and making this work against internal dist git would be problem.  Also,
> for internal purposes it would be very fine to have both RHEL and Fedora
> dist-git support (not RHEL or Fedora only).
> 
> Ideally, Fedora/RHEL dist-gits could be just configurable options, so we
> could
> add Mageia/CentOS dist-git easily too.

Building from DistGit is now supported. We will however need to provide additional distgitX configuration in https://pagure.io/copr/copr/blob/master/f/rpmbuild/main.ini upon requests.
We could add "default distgit" configuration but I don't think it is particularly necessary. Feel free to reopen if you think otherwise.

Comment 6 Pavel Raiskup 2020-05-05 09:20:42 UTC
Eventually moved to https://pagure.io/copr/copr/issue/529


Note You need to log in before you can comment on or make changes to this bug.