Bug 1367220

Summary:	Fedora 25 composes failing due to use of old pungi(?)
Product:	[Fedora] Fedora	Reporter:	Adam Williamson <awilliam>
Component:	pungi	Assignee:	Dennis Gilmore <dennis>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	25	CC:	admiller, dennis, lsedlar, robatino
Target Milestone:	---
Target Release:	---
Hardware:	All
OS:	All
Whiteboard:	AcceptedBlocker
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-08-16 18:56:58 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1277284

Description Adam Williamson 2016-08-15 22:49:17 UTC

Fedora 25 composes are currently failing. I believe this is because pungi-fedora has been updated for the backwards-incompatible change to how 'failable' deliverables are defined:

https://pagure.io/pungi-fedora/c/2ac4a62920900ae8198e2f0fadd2196ed0c57ce3?branch=master

but pungi 4.1.8 - which introduces the change - is not 'stable' for Fedora 25 yet. It has been built:

http://koji.fedoraproject.org/koji/buildinfo?buildID=790649

but it was after the Bodhi activation point, and there is no update submitted. So we need, I believe, to have an update for pungi 4.1.8 submitted (and marked as fixing this bug), and then we need to push it stable so the compose process uses it. Either that, or we need to revert the pungi-fedora change (which would in turn break Rawhide, since 4.1.8 has gone stable there...so in fact we'd need an f25 branch for pungi-fedora so we could diverge configuration between f25 and Rawhide).

The symptom is that the compose process fails as soon as any image fails to compose, as no images are considered 'failable' by the old Pungi, after the configuration changes.

20160815.n.1 compose failed due to the KDE live image compose failing. All the other composes between 20160810.n.0 and 20160815.n.0 failed when an OStree compose attempt failed. If you look here:

https://kojipkgs.fedoraproject.org/compose//branched/Fedora-25-20160810.n.0/logs/x86_64/ostree/ostree-2/runroot.log

you can see the OStree failure actually occurred in 20160810.n.0 too, but did not fail the whole compose. And we know of course that live images often fail to compose due to https://bugzilla.redhat.com/show_bug.cgi?id=1315541 .

This is an automatic Alpha blocker: "Bugs which entirely prevent the composition of one or more of the release-blocking images required to be built for a currently-pending (pre-)release" - this bug prevents the compose of *all* images.

Comment 1 Adam Williamson 2016-08-15 23:04:17 UTC

Ah, so in fact I think the issue was the opposite: the new pungi is being used (apparently it doesn't matter if it was stable), but the f25 branch of pungi-fedora was not updated with the new 'failable' configuration stuff until 7 hours ago:

https://pagure.io/pungi-fedora/c/716f612efd6a8e5e3a6d980f382343169bd8047e?branch=f25

the currently running compose - https://kojipkgs.fedoraproject.org/compose//branched/Fedora-25-20160815.n.2/ - should I *think* be the first run with the updated pungi-fedora config, so we'll see how that goes.

Comment 2 Lubomír Sedlář 2016-08-16 06:25:19 UTC

You can see what configuration was used for a compose in logs/global/config-dump-$date.global.log.

Comment 3 Adam Williamson 2016-08-16 06:41:07 UTC

yes, but we can't (at least, nowhere I could spot) see what version of pungi is used. I think the situation is as described in #c1, though. 20160815.n.2 completed, so I think this is fixed,  but I'll leave it open in case dgilmore wants to chime in.

Comment 4 Lubomír Sedlář 2016-08-16 06:55:02 UTC

You're right, which is exactly why I opened https://pagure.io/pungi/pull-request/364 to add pungi version information to the log.

Comment 5 Dennis Gilmore 2016-08-16 18:56:58 UTC

bugzilla is not an appropriate place for filing issues like this. the composes run on f24 using the f24 build. we have put pungi in the infra builder repo to ensure we use the version we want.

Comment 6 Adam Williamson 2016-08-16 19:13:44 UTC

We need to file release blocking issues on Bugzilla in order to track them. Complete failure to compose for multiple days in a row is clearly a release blocking issue.