Bug 1526721 - [RFE] More Go packaging
Summary: [RFE] More Go packaging
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: go-srpm-macros
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Jan Chaloupka
QA Contact: Fedora Extras Quality Assurance
URL: https://fedoraproject.org/wiki/More_G...
Whiteboard:
Keywords: Improvement
Depends On: 1523779
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-12-16 18:05 UTC by Nicolas Mailhot
Modified: 2018-03-19 20:52 UTC (History)
3 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2018-03-13 23:14:12 UTC


Attachments (Terms of Use)
Proposed technical files (4.90 KB, application/x-xz)
2017-12-16 18:05 UTC, Nicolas Mailhot
no flags Details
Proposed technical files (4.91 KB, application/x-xz)
2017-12-16 19:57 UTC, Nicolas Mailhot
no flags Details
Proposed technical files (5.12 KB, application/x-xz)
2017-12-17 15:00 UTC, Nicolas Mailhot
no flags Details
Proposed technical files (5.43 KB, text/plain)
2017-12-18 13:17 UTC, Nicolas Mailhot
no flags Details
Proposed technical files (5.14 KB, application/x-xz)
2017-12-18 14:40 UTC, Nicolas Mailhot
no flags Details
Patch to activate autodeps on old-style Go packages (980 bytes, patch)
2017-12-18 15:09 UTC, Nicolas Mailhot
no flags Details | Diff

Description Nicolas Mailhot 2017-12-16 18:05:14 UTC
Created attachment 1368847 [details]
Proposed technical files

This are the technical files associated with

https://fedoraproject.org/wiki/More_Go_packaging

which are proposed here for inclusion in go-srpm-macros

The proposal also depends on

https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation

and

https://bugzilla.redhat.com/show_bug.cgi?id=1523779

Comment 1 Nicolas Mailhot 2017-12-16 19:57 UTC
Created attachment 1368872 [details]
Proposed technical files

The autorequires/autoprovides scripts need to be executable, restore permissions lost in rebasing

Comment 2 Nicolas Mailhot 2017-12-17 15:00 UTC
Created attachment 1369031 [details]
Proposed technical files

Handle compatibility via symlinks (not that compat is a good idea long-term)

Comment 3 Nicolas Mailhot 2017-12-18 13:17 UTC
Created attachment 1369502 [details]
Proposed technical files

Be a little more strict for fake symlinked import paths
Un-lua-ify go-compiler glue so it woks again

Comment 4 Nicolas Mailhot 2017-12-18 14:40 UTC
Created attachment 1369538 [details]
Proposed technical files

one-character fix for aliases import paths

Comment 5 Nicolas Mailhot 2017-12-18 15:09 UTC
Created attachment 1369541 [details]
Patch to activate autodeps on old-style Go packages

Comment 6 Jakub Čajka 2017-12-19 08:41:02 UTC
Hello,
  as this seems as quiet significant change, would you mind submit it as pull request on https://src.fedoraproject.org/rpms/go-srpm-macros?

Could you also provide repository for testing this change? For example as COPR(http://copr.fedorainfracloud.org/) repository (with all necessary build root  changes included)?

Comment 7 Nicolas Mailhot 2017-12-19 09:36:26 UTC
Hi Jakub,

A lot of the work is done behind a corporate firewall, on a private git infra with private management rules, without ssh access to the internet and http filtering making pulling highly inconvenient

I'd rather you integrate the files in the go-srpm-macros Fedora public repo as they are, they are few and short, I'll find a way to commit or submit changes later. However I don't intend to change the files anymore, unless you have specific wishes, or optimization ideas

I'll try to setup a copr repo with package build examples (maybe not before january). Not sure how copr handles bootstraping however :) In the meanwhile I can provide an archive with a spec dump if you want. We have ~ 107 new-style specs and ~43 modified old-style specs (though those have lots of problems due to the approximations old-style packaging produced, and it's usually easier to redo them from scratch than investigate the problems. debundling is not finished yet)

Of course not everything is finished there are still some missing deps waiting for packaging, packages that need polishing, and loops to bootstrap.

Usual testing is to scratch sources and packages, re-spectool sources and mass rebuild everything in mock (requiring about 10 rebuild phases right now)

Comment 8 Nicolas Mailhot 2017-12-19 09:49:06 UTC
>and ~43 modified old-style specs (though those have lots of
> problems due to the approximations old-style packaging produced, and it's
> usually easier to redo them from scratch than investigate the problems.
> debundling is not finished yet)

Examples of problems :
* old Fedora spec installs broken example code, which is picked up by auto-requires
* old Fedora spec declares manually requirements that do not match the current state of the shipped code
* old Fedora specs bundles code that need splitting to avoid poisoning the dep engine
* old Fedora tries to simulate import paths that were already legacy in 2014, and confuses itself and other packages
* old Fedora spec thinks it is working due to incomplete deps and requires, stict autodeps/autoreqs reveal holes in the dependency fabrik

Comment 9 Jakub Čajka 2017-12-19 10:02:06 UTC
(In reply to Nicolas Mailhot from comment #7)
> Hi Jakub,
> 
> A lot of the work is done behind a corporate firewall, on a private git
> infra with private management rules, without ssh access to the internet and
> http filtering making pulling highly inconvenient
> 
> I'd rather you integrate the files in the go-srpm-macros Fedora public repo
> as they are, they are few and short, I'll find a way to commit or submit
> changes later. However I don't intend to change the files anymore, unless
> you have specific wishes, or optimization ideas

Ok, could you post git am-able/apply-able patch for the fedora dist-git repository? I haven't tried PR in pagure myself yet, I believe you that they could be bit clunky/hard to understand.

> 
> I'll try to setup a copr repo with package build examples (maybe not before
> january). Not sure how copr handles bootstraping however :) In the meanwhile
> I can provide an archive with a spec dump if you want. We have ~ 107
> new-style specs and ~43 modified old-style specs (though those have lots of
> problems due to the approximations old-style packaging produced, and it's
> usually easier to redo them from scratch than investigate the problems.
> debundling is not finished yet)

You will have to bootstrap it all your self in simiallar fashion you have to do it in Fedora(although for that you mostly need RCM in Fedora space) or mock.
It would be cool to have repo(overriding, providing newer versions of the core macro packages) that could be used for rebuild testing of this change.

> 
> Of course not everything is finished there are still some missing deps
> waiting for packaging, packages that need polishing, and loops to bootstrap.
> 
> Usual testing is to scratch sources and packages, re-spectool sources and
> mass rebuild everything in mock (requiring about 10 rebuild phases right now)

This bit makes me little bit worried, to be honest I have just looked on this change from 10000ft so it might be bit unwarranted.

Comment 10 Nicolas Mailhot 2017-12-19 13:09:49 UTC
I added a COPR here
https://copr.fedorainfracloud.org/coprs/nim/More_Go_Packaging/

Comment 11 Nicolas Mailhot 2017-12-19 13:15:11 UTC
> > Usual testing is to scratch sources and packages, re-spectool sources and
> > mass rebuild everything in mock (requiring about 10 rebuild phases right now)
> 
> This bit makes me little bit worried, to be honest I have just looked on
> this change from 10000ft so it might be bit unwarranted.

That's just an effect of trying to rebuild every spec from scratch (rebuild everything, then re-rebuild everything that failed in the hope a necessary dep was built in the previous phase, continue till build does not progress, retry the same in bootstrap mode for failing builds, if no remaining bootstrap failure rebuild bootstrapped packages in full mode, then if not failure rebuild everything to make sure everything is build against the latest state)

It would be a lot less work if the current Fedora Go packageset was not so obsolete and incomplete.

The number of phases is an effect of the state of the repo, not an effect of the packaging proposal.

Comment 12 Nicolas Mailhot 2018-01-23 16:52:32 UTC
Sent as a PR here:
https://src.fedoraproject.org/rpms/go-srpm-macros/pull-request/1

Comment 13 Nicolas Mailhot 2018-01-25 10:34:01 UTC
Comment on attachment 1369538 [details]
Proposed technical files

Obsolete – use the code in
https://src.fedoraproject.org/rpms/go-srpm-macros/pull-request/1

Comment 14 Nicolas Mailhot 2018-01-25 10:36:02 UTC
Comment on attachment 1369541 [details]
Patch to activate autodeps on old-style Go packages

This patch proved a bad idea as autodeps are too strict for the approximations in many existing old-style Go packages, and it's usually far easier and faster to rewrite them new-style than investigate what is broken in manual deps and shell code.

Comment 15 Fedora Update System 2018-03-05 14:58:17 UTC
go-srpm-macros-2-16.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-eaf0d85684

Comment 16 Fedora Update System 2018-03-05 14:58:27 UTC
go-compilers-1-25.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-10f9ac892e

Comment 17 Fedora Update System 2018-03-05 20:12:15 UTC
go-srpm-macros-2-16.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-eaf0d85684

Comment 18 Fedora Update System 2018-03-05 20:12:18 UTC
go-compilers-1-25.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-10f9ac892e

Comment 19 Fedora Update System 2018-03-07 14:01:39 UTC
go-compilers-1-26.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-cf775ab077

Comment 20 Fedora Update System 2018-03-08 16:19:51 UTC
go-compilers-1-26.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-cf775ab077

Comment 21 Fedora Update System 2018-03-09 19:54:27 UTC
go-compilers-1-27.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-ddca899013

Comment 22 Fedora Update System 2018-03-11 20:58:21 UTC
go-compilers-1-27.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-ddca899013

Comment 23 Fedora Update System 2018-03-13 23:14:12 UTC
go-srpm-macros-2-16.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

Comment 24 Fedora Update System 2018-03-19 20:52:15 UTC
go-compilers-1-27.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.