Bug 1455249 - github webhook fires unnecessary builds
Summary: github webhook fires unnecessary builds
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Copr
Classification: Community
Component: backend
Version: unspecified
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: clime
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-05-24 14:39 UTC by Michael Mráka
Modified: 2017-06-23 18:55 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-06-23 18:55:41 UTC


Attachments (Terms of Use)
webhook content (7.71 KB, text/plain)
2017-06-19 11:19 UTC, Michael Mráka
no flags Details

Description Michael Mráka 2017-05-24 14:39:51 UTC
Description of problem:
Have a package in copr set up to build automatically from github without using --test. I.e. I want to build only released / tagged versions.
The git webhook should fire a build only when new tag is committed.
Otherwise it's wasting power to rebuild already built version.

See e.g https://copr.fedorainfracloud.org/coprs/g/spacewalkproject/nightly-client/package/spacewalk-backend/.

Version-Release number of selected component (if applicable):
Current production version of Copr.

How reproducible:
always

Steps to Reproduce:
1. in copr setup a package built from github
2. do NOT check 'Build with the --test option.'
3. setup a git webhook for the package
3. commit a change into package's git without tagging new version

Actual results:
A package build is triggered.

Expected results:
No build.

Additional info:

Comment 1 clime 2017-05-25 07:38:15 UTC
This is rather a problem of Github not being able to trigger webhook purely only on tag events. I would very much recommend Gitlab that can do it. Apart from that, it is just an amazing system and project import from Github was made _super_ easy. Also, in COPR, we are going to deploy support for Gitlab webhooks today.

...And after writing all this, I looked once more into Github to find out that it can actually do it too (in Webhook settings, there is "Let me select individual events" and then you can pick just "release", instead of "push"). Still, I am very much amazed by Gitlab and I would recommend it to everyone.

On COPR side, this could be done by not rebuilding the same package (of the same version/release) twice but that would make rebuilding of older packages quite a problem.

Comment 2 Michael Mráka 2017-05-25 10:14:41 UTC
> ...And after writing all this, I looked once more into Github to find out
> that it can actually do it too (in Webhook settings, there is "Let me select
> individual events" and then you can pick just "release", instead of "push").
> Still, I am very much amazed by Gitlab and I would recommend it to everyone.

I've changed webhook to 'release' and will see what will happen.
Thank you for pointing it out.

If it turn out to be a correct solution I will change this bug to 'documentation'.

Comment 3 Michael Mráka 2017-05-26 10:35:00 UTC
Well, setting webhook to 'release' in github interface does not work.
It does not trigger a build at all.

Maybe the content of webhook's js is different than expected?

Comment 4 clime 2017-05-26 13:00:03 UTC
Can you try "Create" event instead? It should be the one that generates "ref_type: tag".

Comment 5 Michael Mráka 2017-05-29 12:27:15 UTC
It seems that "create" event triggered a lot of builds from master branch (of different packages) when I tagged a package in a different branch(?!).

Comment 6 clime 2017-05-29 13:11:49 UTC
Yep, the "create" event does currently trigger rebuild of all the packages in the project the webhook is triggered for. It's kind of silly but I don't see a way to infer what packages should be rebuilt based on the info from that event (https://developer.github.com/v3/activity/events/types/#createevent). We looked at it even before in the comments here: https://pagure.io/copr/copr/pull-request/22#. It seems that this behavior cannot be (easily) improved at the moment.

Comment 7 Michael Mráka 2017-05-29 13:45:34 UTC
> Yep, the "create" event does currently trigger rebuild of all the packages in the project

Well, then it's unusable for us (It would rebuild 150+ packages on every tag).


Looking into details of "create" event triggered by tito tag I can see
{
  "ref": "yum-rhn-plugin-2.6.4-1",
  "ref_type": "tag",
...
}
which clearly identifies the package the event belongs to.
So I think it's possible to guess which package should be rebuilt.


And for standard "push" webhook the normal commit looks like
{
  "ref": "refs/heads/master",
  "before": "f5f28f258f8464c9946dacad78083ad4b6dc7df2",
  "after": "5ff694b1bdb8ca7f7bf0ec78dae62e323215499e",
  "commits": [...],
...
}
while "push" for tag it looks like
{
  "ref": "refs/tags/spacewalk-java-2.7.74-1",
  "before": "0000000000000000000000000000000000000000",
  "after": "419d99d030e199ea2f054b9ded1fe25b6507dbd1",
  "commits": [],
...
}
so again it can be distinguished.
I think in case of "--test option" unchecked the "push" events different than ref:"refs/tag/...." should be ignored.

Comment 8 clime 2017-05-29 16:29:06 UTC
Yes, in case Tito is used for release engineering, then it can be distinguished (because it tags with the package name and version always). I will implement this logic and deploy tomorrow.

--test is just a Tito switch for building from HEAD. I cannot get my head around it would do something else.

Comment 9 clime 2017-05-30 14:39:09 UTC
I made a patch (experimental) here: https://pagure.io/copr/copr/c/a26d1c2a48dece7f052aad94cfad69815e582864 and deployed. Please try.

Comment 10 Michael Mráka 2017-06-07 13:39:38 UTC
It still triggers builds for non-tag commits.

Comment 11 clime 2017-06-07 14:58:35 UTC
I am not quite getting what you mean. Have you set sending of "Create" events only? If so, then it should not trigger builds for non-tag commits because no event should be sent in that case.

Comment 12 Michael Mráka 2017-06-16 09:33:54 UTC
Ok, that was a misconfiguration on our side. Webhook was set to "push" not "create".

The "create" webhook seems to work correctly.

There's still a minor issue I don't understand - it fired build https://copr.fedorainfracloud.org/coprs/g/spacewalkproject/nightly/build/565302/
although there was no commit in the package's directory for a month.

Comment 13 clime 2017-06-19 06:52:38 UTC
I can debug this just with the data available currently available but if you could provide the event that launched the build, it would be much easier for me. 
Could you provide the event that executed the build? If not, I'll get one more coffee and try to do it without the event.

Comment 14 Michael Mráka 2017-06-19 08:58:29 UTC
The only webhook around this time is spacewalk-java-2.7.83-1.fc25 tag:
  22e4ce80-510b-11e7-959e-6a07a07f365a 2017-06-14 16:09:53
https://github.com/spacewalkproject/spacewalk/settings/hooks/14126552#details

Comment 15 Michael Mráka 2017-06-19 09:01:31 UTC
And I can see this pattern repeats pretty deterministically :)
https://copr.fedorainfracloud.org/coprs/g/spacewalkproject/nightly/builds/

Comment 16 clime 2017-06-19 10:45:12 UTC
Actually, I can't see the event cause I would need some kind of admin access to see it. Anyway, the mention that it happens deterministically helped as well.

The name of that package is 'spacewalk' and that's appearing in basically every tag created.

Can you, please, post the event here as an attachment? I will fix the package name matching condition to be more specific.

Comment 17 Michael Mráka 2017-06-19 11:19:42 UTC
Created attachment 1289065 [details]
webhook content

Comment 18 Michael Mráka 2017-06-19 11:27:40 UTC
(In reply to clime from comment #16)
> The name of that package is 'spacewalk' and that's appearing in basically
> every tag created.
... 
> Can you, please, post the event here as an attachment? I will fix the
> package name matching condition to be more specific.

Yes, something like 
-                     and package.name not in ref:
+                     and package.name != re.search(r'(.*)-[^-]+-[^-]+$', ref).group(1):
should be it.

Comment 19 clime 2017-06-20 12:16:52 UTC
I applied the patch (approximately) here https://pagure.io/copr/copr/c/13e5c95fe922fb1e56963d4461e9f50458432864. Should be in production this week.

Comment 20 clime 2017-06-23 18:55:41 UTC
New copr-frontend has been deployed.


Note You need to log in before you can comment on or make changes to this bug.